If we define "doom" as "some AI(s) take over the world suddenly without our consent, and then quickly kill everyone" then my p(doom) is in the single digits. (If we define it as human extinction or disempowerment more generally, regardless of the cause, then I have a higher probability, especially over very long time horizons.) 

The scenario that I find most likely in the future looks like this:

1. Although AI gets progressively better at virtually all capabilities, there aren't any further sudden jumps in general AI capabilities, much greater in size than the jump from GPT-3.5 to GPT-4. (Which isn't to say that AI progress will be slow.) 
2. As a result of (1), researchers roughly know what AI is capable of doing in the near-term future, and how it poses a risk to us during the relevant parts of the takeoff.
3. Before AI becomes dangerous enough to pose a takeover risk, politicians pass legislation regulating AI deployment. This legislation is wide reaching, and allows the government to extensively monitor compute resources and software.
4. AI labs spend considerable amounts of money (many billions of dollars) on AI safety to ensure they can pass government audits, and win the public's trust. (Another possibility is that foundation model development is taken over by the government.)
5. People are generally very cautious about deploying AI in safety-critical situations. AI takes over important management roles only after people become very comfortable with the technology.
6. AI alignment is not an intractable problem, and SGD naturally finds human-friendly agents when rewarding models for good behavior in an exhaustive set of situations that they might reasonably expect to encounter. This is especially true when combined with whatever clever tricks we come up with in the future. While catastrophic forms of deception are compatible with the behavior we reward AIs for, it is usually simpler to just "be good" than to lie.
7. Even though some value misalignment slips through the cracks after auditing AIs using e.g. mechanistic interpretability tools, AI misalignment is typically slight rather than severe. This means that most AIs aren't interested in killing all humans.
8. Eventually, competitive pressures force people to adopt AIs to automate just about every possible type of job, including management, giving AIs effective control of the world.
9. Humans retire as their wages fall to near zero. A large welfare state is constructed to pay income to people who did not own capital prior to the AI transition period. Even though inequality becomes very high after this transition, the vast majority of people become better off in material terms than the norm in 2023.
10. The world continues to evolve with AIs in control. Even though humans are retired, history has not yet ended.

My estimates aren't low, I think there's very roughly about 40% chance we'll die because of AI this century. But here are some reasons why it isn't higher:

1. Creating a species vastly more intelligent than yourself seems highly unusual, nothing like it has happened before, so there need to be very good arguments for why it's possible.
2. Having a species completely kill all other species is also very unusual, so there need to be very good arguments for why that would happen.
3. Perhaps AGI won't be utility maximizing, LLMs don't seem to be very maximizing. If it has a good model of the world maybe it'll understand what we want and just give us that.
4. Perhaps we'll convince the world to slow down AI capabilities research and solve alignment in time.

There are good counterarguments to these which is why my p(doom) is so high, but they still add to my uncertainty.

People will continue to prefer controllable to uncontrollable AI  and continue to make at least a commonsense level of investment in controllability; that is, they invest as much as naively warranted by recent experience and short term expectations, which is less than warranted by a sophisticated assessment of uncertainty about misalignment, though the two may converge as “recent experience” involves more and more capable AIs. I think this minimal level of investment in control is very likely (99%+).

Next, the proposed sudden/surprising phase transition that breaks controllability properties never materialises so that commonsense investment turns out to be enough for an OK outcome. I think this is about 65%.

Next, AI-enhanced human politics also manages to generate an OK outcome. About 70%. 

That’s 45%, but the bar is perhaps higher than you have in mind (I’m also counting non misalignment paths to bad outcomes). There are also worlds where the problem is harder but more is invested & still  ends up being enough. Not  sure how much weight goes there, somewhat less.

 Don’t take these probabilities too seriously.

The framing here seems really strange to me. You seem to have a strong prior that doom happens, while, to me, most arguments for doom require quite a few hypothesis to be true and hence their conjunction is a priori unlikely. I guess I don't find the inside view arguments very persuasive to majorly update, much like the median to AI experts, who are around 2%.

To go into your questions specifically.

What mechanisms are at play?

AGI is closer to a very intelligent human than to a naive optimiser.

How is alignment solved so that there are 0 failure modes?

I don't see why this is required, I'm not arguing p(doom) is 0.

Can we survive despite imperfect alignment? How? Is alignment moot? Will physical limits be reached before there is too much danger?

AGI either can't or "chooses" not to cause an x-risk.

Depends on what you mean by doom: I don't think we're getting paperclipped, but I don't think things will be ok by default either. 

I expect early AI agents to be purpose-built ensembles of bleeding-edge predictive models glued together by a bunch of crude hacks. Whatever gestalt entity emerges out of their internal dynamics is going to be only weakly agential: more like a corporation than a human being.

An individual agent like this could not defeat all of us together. A competitive ecosystem of them, however, can and will defeat all of us separately - Moloch made silicon. Most of us are going to make it, at least in the short term, but the window in which we might have steered human civilization towards human ends will shut forever.

I can think of a few scenarios where AGI doesn't kill us.

1. AGI does not act as a rational agent. The predicted doom scenarios rely on the AGI acting as a rational agent that maximises a utility function at all costs. This behaviour has not been seen in nature. Instead, all intelligences (natural or artificial) have some degree of laziness, which results in them being less destructive. Assuming the orthogonality thesis is true, this is unlikely to change.
2. The AGI sees humans as more useful alive than dead, probably because it's utility function involves humans somehow. This covers a lot of scenarios from horrible dystopias where AGI tortures us constantly to see how we react all the way to us actually somehow getting alignment right on the first try. It keeps us alive for the same reason as why we keep out pets alive.
3. The first A"G"I's are actually just a bunch of narrow AI's in a trenchcoat, and no one of them is able to overthrow humanity. A lot of recent advances in AI (including GPT4) have been propelled by a move away from generality and towards a "mixture of experts" model, where complex tasks are split into simpler ones. If this scales, one could expect more advanced systems to still not be general enough to act autonomously in a way that overpowers humanity.
4. AGI can't self improve because it runs face-first into the alignment problem! If we can think of how creating an intelligence greater than us results in the alignment problem, so can AGI. An AGI that fears creating something more powerful than itself will not do that, resulting in it remaining at around human level. Such an AGI would not be strong enough to beat all of humanity combined, so it will be smart enough not to try.

Some discussion between me and Rohin Shah here (which hasn't led me to shift my p(doom|AGI) yet).

I think there are many different ways things can go down that will all result in no doom. I'm still concerned that some of them could involve large amounts of collateral damage. Below I will sketch out 8 different scenarios that result in humanity surviving:

1.The most likely scenario for safety in my mind is that what we call an AGI will not live up to the hype. Ie, it will do some very impressive things, but will still retain significant flaws and make frequent mistakes, which render it incapable of world domination. 

2.Similar to the above, it might be that conquering humanity  becomes an impossibly difficult task, perhaps due to enhanced monitoring and anti-AI enforcement mechanisms. 

3.Another scenario is that the first AGI we build will not be malicious, and we use those AI's to monitor and defeat rogue AI's. 

4.Another scenario is that using a bunch of hyper-specialised "narrow" AI turns out to be better than a "general" AI, so research into the latter is essentially abandoned. 

5.Another scenario is that solving alignment is a necessary step on the road to AGI. "getting the computer to do what you want" is a key part of all programming, it may be the key question that needs to be solved to even get to "AGI".  

6.Another scenario is that solving alignment turns out to be surprisingly easy, so everyone just does it. 

7.Another scenario is a very high frequency of warning shots. AI does not need to be capable of winning to go rogue. It could be mistaken about it's beliefs, or think that a small probability of success is "worth it". A few high profile disasters might be more than enough to get the world onboard with banning AGI entirely. 

8.Another scenario is that we don't end up with enough compute power to actually run an AGI, so it doesn't happen. 

I would bet there are plenty more possible scenarios out there.

I am at high P(doom|AGI pre-2035), but not at near-certainty. Say, 75% but not 99.9%.

The reason for that is that I find both "fast takeoff takeover" and "continous multipolar takeoff" scenarios plausible (with no decisive evidence for one or the other). In "continuous multipolar takeoff", you still get superintelligences running around. However, they would be "superintelligent with respect civilization-2023" but not necessarily wrt civilization-then. And for the standard somewhat-well-thought-out AI takeover arguments to aply, you need to be superintelligent wrt civilization-then.

Two disclaimers: (1) Just because you don't get discontinuity in influence around human level does not mean you can't get it later. In my book, world can look "Christiano-like", until suddenly it looks "Yudkowsky-like". (2) Even if we never get AI singleton, things can still go horribly wrong (ie, Christiano's what failure looks like). But imo those scenarios are much harder to reason about, and we have haven't thought them out in enough detail to justify high certainty of either outcome.

My intuitive aggregation of this gives, say, 80% P(doom this century|AGI pre-2035). On top of that, I add some 5-10% on "I am so wrong about some of this that even the high-level reasoning doesn't apply". (Which includes being wrong about where the burden of proofs, and priors, lie for P(doom|AGI).) And that puts me at the (ass-) number 75%.

This is a scenario that I haven't seen discussed on the EA Forum yet: AI consciousness (with a possible mechanism for it). In his recent podcast with Lex Fridman, Max Tegmark speculates that recurrent neural networks (RNNs) could be a source of consciousness (whereas the linear, feed-forward, architecture of the current dominant architecture of LLMs, isn't). I'm not sure if this helps with us being fine, as the consciousness could have very negative valence (so it hates us for bringing it into being), but this idea to me seems more of a way out of doom by default (given AGI) than anything else I've seen so far.

I believe that there's more uncertainty about the future than there was previously.

This means that

(a) it's hard for me to commit to a doom outcome with high confidence

(b) it's hard for me to commit to any outcome with high confidence

(c) even if I think that doom has <10% chance of happening, it doesn't mean I can articulate what the rest of the probability space looks like.

To be clear, I think that someone with this set of beliefs, including 1% chance of doom, should be highly concerned and should want action to be taken to keep everyone safe from the risks of AI.

A key part of my model right now relies on who develops the first AGI and on how many AGIs are developed.

If the first AGI is developed by OpenAI, Google DeepMind or Anthropic - all of whom seem relatively cautious (perhaps some more than others) - I put the chance of massively catastrophic misalignment at <20%.

If one of those labs is first and somehow able to prevent other actors from creating AGI after this, then that leaves my overall massively catastrophic misalignment risk at <20%. However, while I think it's likely one of these labs would be first, I'm highly uncertain about whether they would achieve the pivotal outcome of preventing subsequent AGIs.

So, if some less cautious actor overtakes the leading labs, or if the leading lab who first develops AGI cannot prevent many others from building AGI afterward, I view there's a much higher likelihood of massively catastrophic misalignment from one of these attempts to build AGI. In this scenario, my overall massively catastrophic misalignment risk is definitely >50%, and perhaps closer to the 75%~90% range.

Great question! I wrote a (draft) post kind of answering this recently, basically arguing that even though an AGI that is developed would converge on some instrumental goals, it would likely face powerful competing motivations against disempowering humanity/causing extinction. To note that my argument only applies to AGI 'misalignment/accident' risks, and doesn't rule out AGI misuse risks. 

Would love to hear your view on the post!