Katja runs AI Impacts and recently published What do ML researchers think about AI in 2022, a new survey of AI Experts. We start this episode by discussing what Katja is currently thinking about, namely an answer to Scott Alexander on slowing down AI Progress, and then go on discussing her survey and other considerations for estimating AI risk.

Below are some highlighted quotes from our conversation (available on Youtube, Spotify, Google Podcast, Apple Podcast). For the full context for each of these quotes, you can find the accompanying transcript.

On Slowing Down AI

The Astral Codex Ten Post Katja Is Replying To

"[Scott] wrote a blogpost about whether we shouldn't or why we shouldn't try and slow down artificial intelligence progress. He's noticing that lately, various people have been saying that maybe we should slow down AI progress if AI progress is a threat to the existence of humanity. He outlines some reasons you might not want to slow down AI progress."

Are we actually in an arms race?

"The first question is, are you actually in an arms race? It seems not obvious to me that the AI situation at present is an arms race or likely to be an arms race. In the classic arms race scenario, it's not the case that when you win the arms race, you get destroyed. In the very simplified version of this arms race, it's quite unlike a usual arms race. The usual arms race is a prisoner's dilemma, basically. You always have an incentive to build more arms regardless of what the other player does. That's not true if you get killed in the end. Then you can build more complicated models, and sometimes you should race perhaps depending on the details of it, but often, not."

Surveying AI Experts in 2022

Forecasting High Level Machine Intelligence

"[High Level Machine Intelligence is] when unaided machines can accomplish every task better and more cheaply than human workers, ignoring things where it's intrinsically advantageous to be a human, for instance, being on a jury [...] We gave [AI Capabilities Researchers] three different probabilities, 10%, 50% and 90%, and we asked them in what years that amount of probability would've been reached basically. And then, to a different set of people, we gave them years and we asked them what the probability was that we would have high level machine intelligence by those years. And then, a different set of people again, we asked about not high level machine intelligence but full automation of labor, which is when AI can do all of the occupations. Which, to my mind is quite similar to when will AI be able to do all of the tasks. But it turned out to be much later."

ML Researchers do consider extremely bad outcomes

"They were all authors at NeurIPS or ICML. We just wrote to 50% of the authors at those conferences [...] I think in the end [the response rate] was maybe roughly 15%. [...] A lot of people who work on AI capabilities do think there are serious concerns with safety. So I think it's very unclear what different people want here, and probably there are a lot of different views. It seems like, among people working on AI capabilities, I guess in this recent survey I worked on, a lot of them said they thought there was more than a 10% chance of an extremely bad outcome."

AI Alignment is increasingly becoming a concern

[...] We describe something the alignment problem and ask, "Do you think this is an important problem? Is it a hard problem? Is it a valuable problem to work on at the moment?" And I think, for all of those answers, the distribution shifted toward it being important and valuable and hard[...] They had five options for how important it is, say. For importance, the top evaluation of importance went from 5% to 20% of people who thought it was the most important. For the value of working on it today, the top category went from 1% to 8%. And for how hard it is, much harder than other things, it went from 9% to 26%.

Other considerations to estimate AI risk

Katja’s Main Source of Optimism 

"I think my main source of optimism is probably that the AI safety community is mistaken about how bad this is. AGI is not going to destroy the world for sort of mundane reasons that other things don't destroy the world. [...] People have different views about how likely doom is. My probability on 'AI destroys the world', is probably something like 7%. So that's a relatively high probability on some other people around being fairly wrong."

Cognitive Labor will be unequally distributed to Agents

"There are two important things happening with sort of human level-ish AI [...] So previously, throughout history, there's been a sort of allotment of cognitive labor per person, sort of like each person has a brain that they can do a certain amount of thinking with [...] So usually, everyone ends up with some decent fraction of what they started out with and things aren't as wildly unequal as they can be for other things. But with AI, there's going to be just a fire hose of new cognitive labor and it's sort of unclear where it will go. It probably won't be distributed equally among people [...] It could be very disruptive for large fractions of it to be going to some particular narrow set of goals or either one person or one company or something like that.

I think the other thing that is sort of happening at the same time is that there are new agents in the world. So far they have most of the creatures around with goals, doing things. Many are humans. There are also animals, but they don't have that much power in the world. With advanced AI, it seems like we'll basically make something sort of like new alien minds. [...] It seems like these two different things will happen at the same time, which means that they'll kind of be combined so that it could be that huge piles of cognitive labor go to these new minds and I think that's where the giant risk is, which the AI safety people are most worried about."