I think that there's a good chance that a leading, careful AI project could be a huge force for good, substantially reducing existential risk

I think the burden of proof should be on the big AI companies to show that this is actually a possibility. Because right now, the technology, as based on the current paradigm, looks like it's fundamentally uncontrollable.

I think at this point, we are not far off this being 

"Should you work at a leading oil company? (including in non-renewables roles)". 

Or even 

"Hans Bethe has just calculated that the chance of the first A-bomb test igniting the atmosphere is 10%; should you work at the Manhattan Project? (including in non-shutting-it-down roles)".

EA has already contributed massively to the safety-washing of the big AI companies (not to mention kicking off and accelerating the race toward AGI in the first place!) I think EAs should be focusing more on applying external pressure now. There are ways to have higher leverage on existential safety by joining (not yet captured) AI governance, lobbying and public campaigning efforts.

As you recognise yourself in your linked post:

the real world has secret information, way more possible strategies, the potential for technological advancements, defections and betrayal, etc. which all favor the more intelligent party.

Also, consider that the AI has ingested ~all the world's information. That, to me, sounds like a huge resource advantage; a huge strategic advantage - it's not just more intelligent, it's more knowledgeable.

It’s somewhat hard to outthink a missile headed for your server farm at 800 km/h.

This actually made me think of the AI launching the missile, and the humans not having time to think (see this or this). The AI will have a huge speed advantage over us - we will basically be like plants or rocks to it.

if you do think the AGI would win, you have to actually prove it

What would count as "proof" to you, short of an actual global catastrophe?

Good to see that you think the ideas should be explored. I think a global moratorium is becoming more feasible, given the UN Security Council meeting on AI, The UK Summit, the Statement on AI risk, public campaigns etc.

Re compute overhang, I don't think this is a defeater. We need the moratorium to be indefinite, and only lifted when there is a global consensus on an alignment solution (and perhaps even a global referendum on pressing go on more powerful foundation models).

Some people in our community have been convinced that an immediate and lengthy AI moratorium is a necessary condition for human survival, but I don't currently share that assessment.

This makes sense given your timelines and p(doom) outlined above. But I urge you (and others reading) to reconsider the level of danger we are now in^[1].

1. ^{^}

   Or, ahem, to rethink your priorities (sorry).

9% - the world is in a singleton state controlled by an unaligned rogue AI acting on its own initiative. ...

1% - all humans are extinct due to an unaligned rogue AI acting on its own initiative

This is interesting and something I haven't seen much expressed within EA. What is happening in the 8% where the humans are still around and the unaligned singleton rogue AI is acting on it's own initiative? Does it just take decades to wipe all the humans out? Are there digital uploads of (some) humans for the purposes of information saving?^[1] Is a ceiling on intelligence/capability hit upon by the AI which means humans retain some economic niches? Is the misalignment only partial, so that the AI somehow shares some of humanity's values (enough to keep us around)?

I think conditional on producing minimal menace AI by the end of 2070, there’s a 28% chance an existential risk would follow within the next 100 years that could be attributed to that AI system.

Does this mean that you think we get alignment by default? Or alignment is on track to be solved on this timeline? Or somehow we survive misaligned AI (as per the above discrepancy between your estimates for singleton unaligned rogue AI and human extinction)? As per my previous comment, I think the default outcome of AGI is doom with high likelihood (and haven't received any satisfactory answers to the question If your AGI x-risk estimates are low, what scenarios make up the bulk of your expectations for an OK outcome?

1. ^{^}

   This still seems like pretty much an existential catastrophe in my book, even if it isn't technically extinction.

Thanks for your detailed answers Peter. Caroline Jenmaire's "minimal menace" is a good definition of AGI for our purposes (but also so is Holden Karnofsky's PASTA, OpenPhil's Transformative AI and Matthew Barnett's TAI.)

I'm curious about your 5% by 2035 figure. Has this changed much as a result of GPT-4? And what is happening in the remaining 95%? How much of that is extra "secret sauce" remaining undiscovered? A big reason for me updating so heavily toward AGI being near (and correspondingly, doom being high given the woeful state-of-the-art in Alignment) is the realisation that there very well may be no additional secret sauce necessary and all that is needed is more compute and data (read: money) being thrown at it (and 2 OOMs increase in training FLOP over GPT-4 is possible within 6-12 months).

the possibility of policy delaying AI development

How likely do you consider this to be, conditional on business as usual? I think things are moving in the right direction, but we can't afford to be complacent. Indeed we should be pushing maximally for it to happen (to the point where, to me, almost anything else looks like "rearranging deckchairs on the Titanic").

Whilst I may not be a professional forecaster, I am a successful investor and I think I have a reasonable track record of being early to a number of significant global trends: veganism (2005), cryptocurrency (several big wins from investing early - BTC, ETH, DOT, SOL, KAS; maybe a similar amount of misses but overall up ~1000x), Covid (late Jan 2020), AI x-risk (2009), AGI moratorium (2023, a few days before the FLI letter went public).

Weird. Does this mean they predicted GPT-4's performance in advance (and also didn't let that update them toward doom)!?

Ok, now Cameron is saying “I think the weaponization of AI is the biggest danger,” he said. “I think that we will get into the equivalent of a nuclear arms race with AI, and if we don't build it, the other guys are for sure going to build it, and so then it'll escalate."

Titanic filmmaker James Cameron agrees with experts that "AI is the biggest danger" to humanity today and claims he warned the world about the issue way back in 1984 in his movie The Terminator. This comes as the so-called 'three godfathers of AI' have recently issued warnings about the need to regulate the quickly evolving technology.

In an interview with CTV News Chief Vassy Kapelos, Cameron said: "I absolutely share their concern," he added: "I warned you guys in 1984, and you didn't listen."

Forecasters stopped updating their forecasts on 31st October 2022

Would be interested to see an update post-chatGPT (esp. GPT-4). I know a lot of people who have reduced their timelines/increased their p(doom) in the last few months.

Re existential security, what are your AGI timelines and p(doom|AGI) like, and do you support efforts calling for a global moratorium on AGI (to allow time for alignment research to catch up / establish the possibility of alignment of superintelligent AI)?