The main goal of my work these days is trying to reduce the chances of individuals or small groups causing large-scale harm through engineered pandemics, potentially civilizational collapse or extinction. One question in figuring out whether this is worth working on, or funding, is: how large is the risk?

One estimation approach would be to look at historical attacks, but while they've been terrible they haven't actually killed very many people. The deadliest one was the September 11 attacks, at ~3k deaths. This is much smaller scale than the most severe instances of other disasters like dam failure, 25k-250k dead after 1975's Typhoon Nina, or pandemics, 75M-200M dead in the Black Death. If you tighten your reference class even further to include only historical biological attacks by individuals or small groups, the one with the most deaths is just five, in the 2001 anthrax attacks.

Put that way, I'm making a pretty strong claim: while the deadliest small-group bio attack ever only killed five people, we're on track for a future where one could kill everyone. Why do I think the future might be so unlike the past?

Short version: I expect a technological change which expands which actors would try to cause harm.

The technological change is the continuing decrease in the knowledge, talent, motivation, and resources necessary to create a globally catastrophic pandemic. Consider someone asking the open source de-censored equivalent of GPT-6 how to create a humanity-ending pandemic. I expect it would read virology papers, figure out what sort of engineered pathogen might be appropriate, walk you through all the steps in duping multiple biology-as-a-service organizations into creating it for you, and give you advice on how to release it for maximum harm. And even without LLMs, the number of graduate students who would be capable of doing this has been increasing quickly as technological progress and biological infrastructure decrease the difficulty.

The other component is a shift in which actors we're talking about. Instead of terrorists, using terror as a political tool, consider people who believe the planet would be better off without humans. This isn't a common belief, but it's also not that rare. Consider someone who cares deeply about animals, ecosystems, and the natural world, or is primarily focused on averting suffering: they could believe that while the deaths of all living people would be massively tragic, it would still give us a much better world on balance. Note that they probably wouldn't be interested in smaller-scale attacks: if it doesn't have a decent chance of wiping out humanity then they'd just be causing suffering and chaos without making progress towards their goals; they're not movie villains! Once a sufficiently motivated person or small group could potentially kill everyone, we have a new kind of risk from people who would have seen smaller-scale death as negative.

Now, these people are not common. There's a trope where, for example, opponents of environmentalism claim that human extinction is the goal, even when most radical environmentalists would see human extinction as a disaster. But what makes me seriously concerned is that as the bar for causing extinction continues to lower, the chances that someone with these views does have the motivation and drive to succeed gets dangerously high. And since these views are disproportionately common among serious engineering-minded folks, willing to trust the moral math, I think some will be the kind of highly capable and careful people who could work in secret for years sustained by a clear conviction that they were doing the right thing.

Fortunately, I think this is a risk we can seriously lower. For example, we should:

• Require biology-as-a-service companies to screen for pathogens and apply anti-money laundering-style customer screening ("KYC").

• Ensure LLMs do not help people kill everyone.

• Verify companies releasing open source LLMs have built them in a way where their safeguards can't be trivially removed.

• Detect stealth pathogens, in a way that would give us warning while there is still time to do something about it (what I'm working on).

• Develop much better and cheaper PPE so that once we detect pandemic we can keep the core functions of society functioning.

• Improve our ability to evaluate new vaccines and other medicines much more quickly so we could potentially roll out a countermeasure in time to stop an in-progress pandemic.

If you want to read more in this direction I'd recommend. Kevin Esvelt's 80,000 Hours podcast appearance (transcript) and his Delay, Detect, Defend paper.