A few days ago I talked to Michael Littman, a CS professor at Brown working in Machine Learning and Artificial Intelligence, as part of my project of assessing superintelligence risk. Michael doesn't think that work on long-term AI risk is valuable, and I wanted to get a better sense of why. In the same fashion as my conversation with Dario Amodei, here's a summary of what I learned from him.

Michael believes that we'll probably have machines that are as smart as humans (AGI, for short) at some point. He doesn't think it's certain, and he's really interested in the question of whether human-style intelligence is achievable, but he thinks we'll probably figure it out eventually. At the same time, he sees this as very far off. Not just "far" in the sense of a long time coming, though he does think that's probably the case, but "far" in the sense of how much more technical and conceptual work is required before we get there.

For example, I asked him what he thought of the idea that to we could get AGI with current techniques, primarily deep neural nets and reinforcement learning, without learning anything new about how intelligence works or how to implement it ("Prosaic AGI" [1]). He didn't think this was possible, and believes there are deep conceptual issues we still need to get a handle on. He's also less impressed with deep learning than he was before he started working in it: in his experience it's a much more brittle technology than he had been expecting. Specifically, when trying to replicate results, he's often found that they depend on a bunch of parameters being in just the right range, and without that the systems don't perform nearly as well.

The bottom line, to him, was that since we are still many breakthroughs away from getting to AGI, we can't productively work on reducing superintelligence risk now.

He told me that he worries that the AI risk community is not solving real problems: they're making deductions and inferences that are self-consistent but not being tested or verified in the world. Since we can't tell if that's progress, it probably isn't. I asked if he was referring to MIRI's work here, and he said their work was an example of the kind of approach he's skeptical about, though he wasn't trying to single them out. [2]

I asked him what he thought of Concrete Problems in AI Safety (pdf), and Michael told me he hadn't read it in detail but had skimmed it. His response was that getting better at specifying what we care about is a great research problem, and an area he's working on. It's technology that would be useful to us today, and he's happy to see it developed better. He doesn't see it, however, as useful from a long-term AI safety perspective because we just don't know anything about what AGI would look like yet.

At this point, I was trying to figure out if the key disagreement between him and, say, Dario or Paul was on whether Prosaic AGI was possible. I asked what he would think we should work on if he believed that all we needed to get to AGI was a lot more engineering work on current technologies without discovering anything fundamentally new. Unfortunately, this was far enough from what he actually believes that he found it really hard to get into the hypothetical. He thought that even in this hypothetical he probably wouldn't think "loss of control" issues were among the most urgent ones we have, but he really wasn't very sure what he would think if he were actually in that situation.

[1] I'm not sure whether Paul Christiano coined this term, but he's who I've heard it from. For more detail on it, see the first section of Prosaic AI Alignment.

[2] He followed up by email: "At root, I feel that the AI risk community is trying to extrapolate from our cartoonish current understanding of what intelligence is and how it works and making a deeply unsubstantiated leap that more raw computation will give us a great deal more of whatever is powerful about intelligence. This kind of extrapolation has been very successful in areas like physics, where we can infer the structure of far away galaxies and project the impact of what it means for more energy or gravity or temperature to be concentrated in one place. I think it's a mistake to think of intelligence as a physical quantity and to put any faith into similar extrapolations in the cognitive realm. It's simply too easy for us to misinterpret our own introspections about intelligence and the science of the production of creative new ideas is still very very young."