thebench.bio #1 Ecosystem for the Bio-Tech Industry
All Model Evaluation questions

Question 11 of 20 Fundamentals Interpretation

Why should every model be compared with a simple baseline such as a dummy classifier?

Short answer

A score means little until you know what a trivial rule achieves on the same data. Always predicting the majority class can give 95 percent accuracy when 5 percent of samples are positive, so a model's 96 percent may add almost nothing. A baseline shows how much of the score comes from the model.

Why

The scikit-learn guide calls comparing an estimator with simple rules of thumb a basic sanity check, and DummyClassifier implements several: most_frequent always predicts the most common label, prior predicts the class proportions, stratified guesses at random in proportion to the classes. DummyRegressor predicts the mean or median. Run one under the same cross-validation as the real model.

In a dataset of 2,000 compounds with 100 actives, most_frequent scores 95 percent accuracy and an AUC of 0.5. A random forest reporting 96 percent accuracy has learned very little, while its AUC and precision at the top of the ranking tell the real story. Baselines also catch bugs.

A model that cannot beat a dummy suggests leakage was avoided but signal is absent, or that labels are misaligned with features. A model that beats it by an implausible margin suggests leakage. Beyond dummies, a strong domain baseline is often more informative, such as logistic regression on age and stage for a clinical outcome, or nearest-neighbor similarity for compound activity.

The question an audience cares about is how much the new model adds over what they already have, not over random guessing.

A strong answer covers

  • Scores need a reference point
  • Majority-class accuracy equals the majority share
  • DummyClassifier strategies in the same CV
  • Baselines expose bugs and leakage
  • Strong domain baseline is the real comparator

Common mistakes

  • Reporting high accuracy on imbalanced data without stating the majority-class rate
  • Comparing only against a random baseline when a simple clinical score already exists