Design a validation scheme for an ML model intended to predict outcomes in future calendar periods.

Why

Clinical practice, disease prevalence, coding and measurements drift over time. Training with future patients or future-updated features lets the model access information that would not exist at deployment. Repeated visits create additional group dependence.

How to work it through

1. Specify an index date and prediction horizon.
2. Train on an earlier period, tune on a later but still pre-test period, and reserve the most recent untouched period for evaluation.
3. Keep patient episodes appropriately grouped, implement feature cutoffs and gap or purge periods where outcome windows overlap.
4. Simulate periodic retraining prospectively, monitor calibration and report performance by calendar cohort and site.

Where the answer stops holding

Severe seasonal effects may require multiple rolling-origin windows. Patient reappearance is sometimes legitimate for an established-patient use case, but the target population and leakage control must be explicit.