What does end-to-end provenance mean for a research dataset?

Short answer

End-to-end provenance links each released value to its raw origin, transformations, reference resources, quality decisions and responsible pipeline or person.

Why

Model lineage as a graph of entities, activities and agents instead of just an output folder. For a harmonized genomics dataset, one record may depend on original instrument files, base-calling software, a reference genome, an annotation release and manual QC decisions. Store immutable source identifiers, run IDs, code hashes, parameters, validation reports and accession or DOI information where appropriate.

Trace lineage forward to find outputs affected by a faulty instrument lot, and backward to justify a published figure. Access-controlled raw data may require a pointer rather than public exposure of individual-level data. Provenance supports reproducibility and accountability but does not itself prove that the measurement or scientific interpretation was correct.

Common mistakes

Keeping the final CSV and a README while deleting the input-to-output transformation mapping

The interviewer might ask next

How would you identify every derived table affected by a recalled sequencing reference build?