How would you process a biological dataset too large for memory?

Why

First determine whether the pipeline truly requires all rows or just selected columns, samples or summary statistics. For a huge assay table, read chunks with explicit dtypes and compute mergeable per-group aggregates instead of concatenating every chunk. Partitioned Parquet can support column projection and filtering.

Dask can schedule out-of-core operations when partition sizes are safe for worker memory. Beware expensive shuffles and joins, duplicate identifiers and group-level statistics that cannot be computed correctly by simply averaging chunk means. Use checksums, row counts and tests on a known small reference dataset.

Record processing order when numerically sensitive reductions make exact floating-point agreement important. More parallelism does not help if each worker allocates a full input copy.