thebench.bio #1 Ecosystem for the Bio-Tech Industry
All Machine Learning questions

Question 15 of 20 Intermediate Concept

How does a random forest differ from gradient boosting?

Short answer

A random forest grows many deep trees independently on bootstrap samples with random feature subsets and averages them, which mainly reduces variance. Gradient boosting grows shallow trees one after another, each correcting the errors of the ones before, which mainly reduces bias and usually needs more careful tuning.

Why

The scikit-learn ensemble guide describes the contrast directly. In a random forest each tree is built from a bootstrap sample of the training data and considers a random subset of features at each split. The trees are deep and overfit individually, but their errors are partly independent, so averaging them gives a stable prediction.

Forests are hard to badly misconfigure, and more trees rarely hurt except in time. Gradient-boosted trees work sequentially. Each new tree is fitted to the remaining errors of the ensemble so far, and a learning rate controls how much each tree contributes.

The guide notes that boosting uses shallow trees that underfit individually and builds them in sequence, and that histogram-based gradient boosting is often a competitive alternative to forests, fast on large tables and able to handle missing values natively. Because boosting keeps reducing training error, too many trees or too high a learning rate will overfit, so it relies on early stopping and tuning. For a tabular assay dataset, both are reasonable starting points after a linear baseline.

On small biological datasets the gap between them is often smaller than the gap created by a leaky validation scheme, so validation design matters more than the choice.

A strong answer covers

  • Forest: independent deep trees, bootstrap, feature subsets
  • Boosting: sequential shallow trees correcting errors
  • Forest reduces variance, boosting reduces bias
  • Boosting needs learning rate, early stopping, tuning
  • Histogram boosting handles missing values natively

Common mistakes

  • Believing a random forest cannot overfit because it averages many trees
  • Adding boosting rounds without early stopping on a validation set