Benchmarking Bowtie2 Threading

December 1st, 2023
bio, tech
I've been using Bowtie2 to align reads to genomes, and one of it's many settings is the number of threads. While sometimes people advise using about as many threads as your machine has cores, but if I'm running on a big machine are there diminishing returns or a point at which more threads are counterproductive? Am I better off running more samples in parallel with more threads each, or fewer with fewer?

I decided to run a few tests on an AWS EC2 with a c6a.8xlarge 32-core AMD machine. The test consisted of running one 7.2Gb 48M read-pair sample (SRR23998356) from Crits-Christoph et. al 2021 through Bowtie2 2.5.2 with the "Human / CHM13plusY" database from Langmead's Index Zone. The files were streamed from AWS S3 and decompressed in a separate process. See the script for my exact test harness and configuration.

What I found (sheet) was that initially allocating additional threads helps a lot, but after ~8 it was plateauing and after ~10 more threads were very slightly starting to hurt:

Compare what I saw to the best you could hope for, where the task is perfectly parallelizable across threads:

This shows that initially Bowtie2 makes very efficient use of additional threads but, as you'd expect from the plateauing, after ~8 the benefit drops off.

Another way to look at this is to chart how many thread-seconds it takes to process the whole dataset:

On this plot perfect parallelism would be a horizontal line, and it's pretty close to that up through 8 threads. After that, however, it starts rising much more steeply, showing much less efficient use of threads.

You might wonder how much of this is Bowtie2, and how much is my test environment. For example, perhaps by the time we get to eight threads we're at the limit of how quickly this machine can download from S3 and decompress the data? Bowtie2 can't work any faster than the data arrives! I ran a "null" version of this where I replaced bowtie2 with paste (test script) and found that this consistently took 44-45s. Since it's all streaming and the fastest run took 189s (4x) I don't think this was having an appreciable effect.

Overall, it looks like even on a very large machine you shouldn't set Bowtie2's --threads above 10, and unless you have idle cores 8 is probably a better value.

Comment via: facebook, lesswrong, mastodon, substack

Recent posts on blogs I like:

Americans: call your senators today to stop the Save Our Bacon Act

The Farm Bill currently under consideration by the U.S.

via Thing of Things May 7, 2026

Fiddle Practice

For a while I wasn't learning how to play violin very well because whenever it was time to practice I didn't want to. I didn't really like practicing, because (1) it's boring, (2) I have better things to do, and (3) actually I guess there …

via Anna Wise's Blog Posts May 3, 2026

New Pony

I have a new pony. I got her from the Goodwill. Her name is "Rainbow" and her nickname is "Sparkles". We went the usual way, in the pink stroller. We went downstairs and I played with some of the toys. I found the pony in the toy area. …

via Nora Wise's Blog Posts May 3, 2026

more     (via openring)