thebench.bio #1 Ecosystem for the Bio-Tech Industry
All Python for Scientific Computing questions

Question 6 of 6 Advanced Scenario

How would you version a dataset and its preprocessing steps?

Short answer

Give raw data an immutable identifier, version transformation code and parameters, and record a lineage manifest linking every derived table back to its inputs.

Why

A meaningful dataset version includes more than a file name ending in v3. Assign stable identifiers or content hashes to raw files and record collection metadata, consent and access constraints as appropriate. Track the code commit, pinned environment, transformation configuration, filtering criteria and output hashes for each processing run.

A biology example is a gene-count matrix where annotation release, sample exclusions and low-count thresholds can change which genes remain. Store transformations as ordered, executable steps and keep raw observations untouched. A semantic version or release label is useful for users, but reproducibility depends on the underlying provenance.

Test whether recomputing from a historical manifest reproduces the published row counts and QC summaries. Versioning alone does not make a dataset FAIR or scientifically representative.

Common mistakes

  • Overwriting processed.csv repeatedly without recording the raw source, reference release or exclusion logic