How can regression to the mean make a before-and-after study look like the treatment worked?

Why

Every measurement is a true value plus variation, from biology and from the assay. Selecting people or samples whose first reading was very high picks out many whose noise happened to be high that day. On a second reading the noise is fresh, so their average falls toward the population mean.

Barnett, van der Pols and Dobson describe this as natural variation that looks like real change, and note that the effect grows with measurement error and with weaker correlation between repeated measurements. Suppose 500 volunteers are screened and the top 50 by LDL cholesterol receive a supplement. Six weeks later their mean has fallen by 0.4 units.

Some or all of that fall would have happened anyway. Troubleshooting such a result means asking how the subjects were selected, how noisy the measurement is and whether anything untreated was measured the same way. The strongest fix is design: randomize eligible subjects to treatment or control, so both groups regress equally and the difference between them estimates the effect.

Taking several baseline measurements and averaging them reduces the problem, and analysis of covariance with baseline as a covariate is the usual analysis. The same effect appears in labs, for example when only plates with extreme controls are rerun.