Your validation metric changes noticeably between runs with different random seeds. What should you report?

Why

Random seeds control data splits, weight initialization, data shuffling, dropout and subsampling in tree ensembles. The scikit-learn guide on controlling randomness explains that estimators and cross-validation splitters draw randomness through random_state, and that we want an estimator to be robust to its random initialization, not just accurate for one draw. Bouthillier and colleagues showed that variance from data sampling, initialization and hyperparameter choices markedly affects benchmark conclusions.

If a model scores AUC 0.78, 0.74, 0.80, 0.75 and 0.77 across five seeds, reporting 0.80 is selection. The honest summary is a mean of about 0.77 with a range of 0.74 to 0.80. To diagnose, vary one source at a time: fix the split and vary initialization, then fix initialization and vary the split.

Large split-driven variance points to a small or heterogeneous dataset, while initialization-driven variance in a neural network suggests unstable training that a lower learning rate or more data may reduce. Fixing a seed makes a run repeatable, which is good for debugging, but it does not make the result more true. When comparing models, use the same seeds and splits for both and compare paired differences.