thebench.bio #1 Ecosystem for the Bio-Tech Industry
All Sequence Alignment questions

Question 18 of 20 Advanced Practical

How would you design a benchmark to compare read aligners on sequences containing long indels?

Short answer

Combine simulated reads with indels of known position and length and real reads from a reference sample with a curated truth set, such as the Genome in a Bottle insertion and deletion benchmark. Score alignment and downstream calls stratified by indel length and sequence context, with a comparison method that handles different representations.

Why

Aligners often handle small indels well and long ones poorly, because a long gap costs more than soft-clipping the read end or splitting it. A good benchmark makes that visible. Simulated data give exact truth for every read, so you can measure whether each read was placed and gapped correctly across indel lengths, from a few bases to beyond read length.

Simulation is idealized, so real data are also needed. The Genome in a Bottle benchmark by Zook and colleagues covers sequence-resolved insertions and deletions of 50 bp or more, 5,262 insertions and 4,095 deletions across 2.51 Gbp of a reference genome. It excludes regions with more than one structural variant, mostly in tandem repeats, because comparison tools cannot yet handle them.

The comparison itself is tricky, because one indel in a repeat can be written at several positions. The GA4GH benchmarking work stresses representation-aware comparison and stratification by variant type and genome context. Report recall and precision per indel length bin and per context, separately for alignment and for the variant calls built on it.

How to work it through

  1. Define the indel length range and contexts that matter for the intended use.
  2. Simulate reads from a genome with inserted indels of known size and position, including repeats.
  3. Add real reads from a reference sample with a curated insertion and deletion truth set.
  4. Align with each candidate aligner using documented settings.
  5. Score read placement on simulated data and variant recall and precision on real data with a representation-aware tool.
  6. Stratify results by indel length and sequence context and report where each aligner fails.

An example

A lab compares two aligners for 150 bp reads. On simulated data, both place over 98 percent of reads with deletions under 20 bp correctly, but for 40 to 60 bp deletions aligner A soft-clips 45 percent of reads while aligner B gaps them correctly in 80 percent. On the GIAB sample, B recovers 71 percent of benchmark deletions of 50 to 100 bp against 52 percent for A, with similar precision.

Both perform poorly in tandem repeats, which the report states as a shared limitation.

Where the answer stops holding

Simulated reads miss real error profiles and library biases. Truth sets exclude the hardest regions, so results overstate performance in tandem repeats. A benchmark on one genome and one read length may not transfer to other species or technologies.

A strong answer covers

  • Use both simulated reads and a curated real truth set
  • Stratify by indel length and sequence context
  • Use representation-aware comparison for indels in repeats
  • Score alignment and downstream calls separately
  • State regions the truth set excludes

Common mistakes

  • Benchmarking only on simulated reads with ideal errors
  • Reporting one overall accuracy figure across all indel sizes
  • Matching indels by exact position and missing equivalent representations