How would you profile and improve memory use before moving a Python analysis to a cluster?

Short answer

Measure peak memory on a representative input first, find which lines allocate the most, then load fewer columns, use smaller dtypes and process in chunks. Only then set the cluster memory request, from measured peak plus margin.

Why

Cluster schedulers kill a job that exceeds its memory request, and a request that is far too high wastes allocation and queues longer. Guessing from file size is unreliable, because a CSV can need several times its size on disk once loaded into a DataFrame, especially when it has text columns stored as Python objects. The pandas scaling guide lists the main levers in order of effort: load less data, use efficient dtypes, process in chunks, and only then reach for an out-of-core library such as Dask.

Each lever changes a different term in peak memory. Fewer columns shrink the table, category dtypes store each distinct label once with small integer codes, and chunking caps peak memory at roughly one chunk plus the running result. Measurement comes first, then reduction, then sizing, and the measurement is repeated after every change so the gain is known rather than assumed.

How to work it through

Run the analysis on a representative subset, such as 10 percent of samples, and record peak memory.
Use tracemalloc snapshots or a line profiler to find the lines that allocate most.
Load only the needed columns with usecols or columns, and give each column an explicit, compact dtype.
Convert repeated text such as gene names or plate IDs to the category dtype, and downcast numeric columns where the range allows.
Replace load-everything steps with chunked or partitioned processing that keeps only running aggregates.
Rerun the profile, check peak memory scales as expected with input size, and set the job request from that peak plus a safety margin.

An example

A variant summary script reads a 6 GB table and peaks at 31 GB, so it fails on 16 GB nodes. memory_usage(deep=True) shows three text columns with a few thousand distinct values take 70 percent of the memory. Converting them to category and reading only 12 of 40 columns brings peak to 7 GB on the full file. Reading per chromosome and combining counts lowers it to 2.5 GB.

The job is submitted with a 4 GB request and completes.

Where the answer stops holding

tracemalloc reports memory allocated through Python's allocator, so total process memory from the operating system can be higher, and it is worth checking peak resident memory as well. Downcasting floats to float32 trades precision for memory and can change results. Chunking works for aggregates that combine cleanly, such as sums and counts, but not directly for medians or global ranks.

Memory on the cluster may also differ because of other libraries and parallel workers each holding copies.

A strong answer covers

Measure peak memory on a representative input
Locate the allocating lines before optimizing
Load fewer columns with compact dtypes and categories
Chunk where aggregates combine cleanly
Size the job request from measured peak plus margin

Common mistakes

Setting the job memory request from the input file size on disk
Downcasting measurement columns to float32 without checking the effect on results
Parallelizing a memory-bound job so each worker loads its own full copy of the data

The interviewer might ask next

How would you compute a median across chunks that cannot all fit in memory?