By the way, I feel now that my first reply in this thread was needlessly snarky, and am sorry about that.

I think there are a bunch of things that make expertise more valuable in AI forecasting than in geopolitical forecasting:

• We've all grown up with geopolitics and read about it in the news, in books and so on, so most of us non-experts already have passable models of it. That's not true with AI (until maybe one year ago, but even now news don't report on technical safety).
• Geopolitical events have fairly clear reference classes that can give you base rates and so on (and this is a tool available to both experts and non-experts) -- this is much harder with AI. That means the outside view is less valuable for AI forecasting.
• I think AI is really complex and technical, and especially hard given that we're dealing with systems that don't yet exist. Geopolitics is also complex, and geopolitical futures will be different from now, but the basic elements are the same. I think this also favors non-experts when forecasting on geopolitics.

And quoting Peter McCluskey, a participating superforecaster:

The initial round of persuasion was likely moderately productive. The persuasion phases dragged on for nearly 3 months. We mostly reached drastically diminishing returns on discussion after a couple of weeks.

[...]

The persuasion seemed to be spread too thinly over 59 questions. In hindsight, I would have preferred to focus on core cruxes, such as when AGI would become dangerous if not aligned, and how suddenly AGI would transition from human levels to superhuman levels. That would have required ignoring the vast majority of those 59 questions during the persuasion stages. But the organizers asked us to focus on at least 15 questions that we were each assigned, and encouraged us to spread our attention to even more of the questions.

[...]

Many superforecasters suspected that recent progress in AI was the same kind of hype that led to prior disappointments with AI. I didn't find a way to get them to look closely enough to understand why I disagreed.

My main success in that area was with someone who thought there was a big mystery about how an AI could understand causality. I pointed him to Pearl, which led him to imagine that problem might be solvable. But he likely had other similar cruxes which he didn't get around to describing.

That left us with large disagreements about whether AI will have a big impact this century.

I'm guessing that something like half of that was due to a large disagreement about how powerful AI will be this century.

I find it easy to understand how someone who gets their information about AI from news headlines, or from laymen-oriented academic reports, would see a fair steady pattern of AI being overhyped for 75 years, with it always looking like AI was about 30 years in the future. It's unusual for an industry to quickly switch from decades of overstating progress, to underhyping progress. Yet that's what I'm saying has happened.

[...]

That superforecaster trend seems to be clear evidence for AI skepticism. How much should I update on it? I don't know. I didn't see much evidence that either group knew much about the subject that I didn't already know. So maybe most of the updates during the tournament were instances of the blind leading the blind.

Scott Alexander points out that the superforecasters have likely already gotten one question pretty wrong, having a median prediction of the most expensive training run for 2024 of $35M (experts had a median of $65M by 2024) whereas GPT-4 seems to have been ~$60M, though with ample uncertainty. But bearish predictions will tend to fail earlier than bullish predictions, so we'll see how the two groups compare in the next years, I guess.

Of course, there could be plenty of other reasons as well, but I thought these two were interesting to highlight.

Yes, for example maybe the AI experts benefited from their expertise on AI.

Thanks for providing the arguments commonly given for and against various cruxes, that's super interesting.

These two arguments for why extinction would be unlikely

• The logistics would be extremely challenging.
• Millions of people live very remotely, and AI would have little incentive to pay the high costs of killing them.

make me wonder what the forecasters would've estimated for existential risk rather than extinction risk (i.e. we lose control over our future / are permanently disempowered, even if not literally everyone dies this century). (Estimates would presumably be somewhere between the ones for catastrophe and extinction.)

I'm also curious about why the tournament chose to focus on extinction/catastrophe rather than existential risks (especially given that its called the Existential Risk Persuasion Tournament). Maybe those two were easier to operationalize?

That's a nice example!

I mention a few other instances of early animal welfare concern in this post:

• The parliament of Ireland passed one of the first known animal welfare laws in 1635. Massachusetts passed one in 1641.
• Margaret Cavendish (1623-1673) "articulate[d] the idea of animal rights"; she wrote: "As for man, who hunts all animals to death [...] is he not more cruel and wild than any bird of prey?"
• Anne Finch (1631-1679) argued against the mechanistic view of animal nature, writing that animals had "knowledge, sense, and love, and divers other faculties and properties of a spirit".
• In 1751, the artist William Hogarth made four engravings that showed a boy torturing animals and gradually becoming a thief and a murderer of humans.
• In 1776 Humphrey Primatt published A Dissertation on the Duty of Mercy and Sin of Cruelty to Brute Animals.
• And then there's Jeremy Bentham's Principles of Morals and Legislation (1780).

Curiously, lots of them seem to come from the Anglo-Saxon sphere (though there's definitely selection bias since I looked mostly through English-speaking sources; also, we have older examples of concern for animals by e.g. Jains and Buddhists).

I think a bunch of people were hesitant about AI safety entering the mainstream because they feared it would severely harm the discussion climate around AI safety (and/or cause it to become a polarized left/right issue).

It used to be done by just typing the @ symbol followed by the person's name, but that doesn't seem to work anymore.

Re scaling current methods: The hundreds of billions figure we quoted does require more context not in our piece; SemiAnalysis explains in a bit more detail how they get to that number (eg assuming training in 3mo instead of 2 years).

That's hundreds of billions with current hardware. (Actually, not even current hardware, but the A100 which is last-gen; the H100 should already do substantially better.) But HW price-performance currently doubles every ~2 years. Yes, Moore's Law may be slowing, but I'd be surprised if we don't get another OOM improvement in price-performance during the next decade, especially given the insatiable demand for effective compute these days.

We don't want to haggle over the exact scale before it becomes infeasible, though---even if we get another 2 OOM in, we wanted to emphasize with our argument that 'the current method route' 1) requires regular scientific breakthroughs of the pre-TAI sort, and 2) even if we get there doesn't guarantee capabilities that look like magic compared to what we have now, depending on how much you believe in emergence. Both would be bottlenecks.

Yeah, I agree things would be a lot slower without algorithmic breakthroughs. Those do seem to be happening at a pretty good pace though (not just looking at ImageNet, but also looking at ML research subjectively). I'd assume they'll keep happening at the same rate so long as the number of people (and later, possibly AIs) focused on finding them keeps growing at the same rate.

Adding to that, there's also the Postmortems & retrospectives tag. There there's e.g. Lessons learned from Tyve, an effective giving startup, Why Anima International suspended the campaign to end live fish sales in Poland and a similar question, How many EAs failed in high risk, high reward projects?

Oh thanks, I misunderstood that and have edited my comment.