Question 20 of 20 Advanced Practical
How would you benchmark a germline variant-calling pipeline against a truth set?
Short answer
Sequence a reference sample with a characterized truth set, such as a Genome in a Bottle genome, compare calls with a tool that handles different representations of the same variant, and report precision and recall within the high-confidence regions, stratified by variant type and genome context.
Why
A benchmark needs a truth set, a fair comparison and honest reporting. The GA4GH Benchmarking Team, in work by Krusche and colleagues, set out best practices. Comparison is harder than it sounds, because one haplotype can be written as different combinations of REF and ALT alleles.
A naive position match then counts the same variant as both a false positive and a false negative. Tools such as hap.py and vcfeval compare haplotypes rather than text. Metrics should be the standard true positives, false positives and false negatives, with precision and recall derived from them, and genotype errors counted distinctly.
Truth sets cover only high-confidence regions, and calls outside them cannot be scored, so a benchmark says nothing about the hardest parts of the genome unless it says so. Stratification matters because a single genome-wide number hides weakness. GIAB stratifications split results by contexts such as coding regions, low mappability, homopolymers and segmental duplications, where performance differs widely.
How to work it through
- Choose a truth set and genome build that match the pipeline, for example a GIAB sample on GRCh38.
- Sequence the reference material with the same wet-lab process used for real samples.
- Run the pipeline unchanged and compare calls to truth with a haplotype-aware tool such as hap.py.
- Restrict scoring to the truth set's high-confidence regions and record their size.
- Report precision and recall for SNVs and indels separately, stratified by genome context.
- Repeat across runs to measure reproducibility, and investigate the largest error classes.
An example
A lab benchmarks its exome pipeline on a GIAB sample. Overall SNV recall in high-confidence regions is 99.6 percent and precision 99.4 percent. Indel recall is 96 percent, and stratification shows it falls to 82 percent in homopolymers longer than 10 bases.
The lab adds a homopolymer caveat to its reports and tests a caller setting that raises recall there to 90 percent without lowering precision.
Where the answer stops holding
High-confidence regions exclude the hardest sequence, so results overstate genome-wide accuracy. A reference cell line DNA does not test sample-specific problems such as FFPE damage or low input. Germline benchmarks do not transfer to somatic or low-fraction detection.
A strong answer covers
- Use characterized reference material such as GIAB
- Haplotype-aware comparison handles representation differences
- Report precision and recall within high-confidence regions
- Stratify by variant type and genome context
- Run the real wet-lab process, not only the software
Common mistakes
- Matching calls by position and allele text only
- Quoting one genome-wide accuracy figure without stratification
- Benchmarking only the software on downloaded reads and ignoring the wet lab
The interviewer might ask next
How would you extend this benchmark to regions outside the truth set's high-confidence calls?