Question 13 of 20 Fundamentals Concept

What is the difference between model parameters and hyperparameters?

Short answer

Parameters are learned from the training data during fitting, such as regression coefficients or tree split points. Hyperparameters are set before fitting and control how learning happens, such as regularization strength, tree depth or learning rate, and are chosen by validation rather than by the fit itself.

Why

The scikit-learn guide defines hyperparameters as parameters that are not directly learned within estimators, passed as constructor arguments, with examples such as C, kernel and gamma for a support vector classifier. A random forest learns its split features and thresholds from data, while the number of trees, max_depth and max_features are hyperparameters. Hyperparameters cannot be fitted on the training loss, because that would always favor the most flexible setting.

A tree with unlimited depth always fits training data best. Instead they are chosen by comparing validation performance across candidates, using GridSearchCV or RandomizedSearchCV with cross-validation. That selection is itself a form of fitting.

If the same folds are used to choose hyperparameters and to report performance, the reported score is optimistic, which is why a separate test set or nested cross-validation is needed. Some choices sit in between. The random seed is not tuned but affects results, and preprocessing choices such as the number of selected features behave like hyperparameters and belong in the search.

Record the final hyperparameters and the search space, because a model cannot be reproduced from its parameters alone.

A strong answer covers

Parameters learned during fit
Hyperparameters set before fit
Hyperparameters chosen by validation, not training loss
Tuning is fitting, so report on separate data
Record the search space and final values

Common mistakes

Choosing hyperparameters by training accuracy
Reporting the best cross-validation score from a grid search as the expected performance on new data

The interviewer might ask next

Why does the best score from GridSearchCV overestimate performance on new data?