3.5 will be reasonably well distributed between the wrong answer (no) and the right answer + a routing that passes directly through NH. My single Poe  GPT-4 fell into the second category.

Particular ChatGPT failure mode that I am wondering if GPT-4 passes: routing questions (the ones I tried "can I drive from Boston to Portland, Maine without passing through New Hampshire", "I want to look at the Arctic Ocean from behind my windshield. Can I do this?" ChatGPT was able to answer both <1/10 times). Anyone with access want to try this?

Certainly agree that we are learning right now (I hope :)). 

"this is only a small fraction of what a human brain does in fact learn"

Disagree here. The description size of my brain (in CS analogy, the size of the circuit) seems much much larger than the total amount of information I have ever learned or ever will learn (one argument: I have fewer bits of knowledge than Wikipedia, describing my brain in the size of Wikipedia would be an huge advance in neuroscience). Even worse, the description size of the circuit doesn't (unless P=NP)  provide any nontrivial bound on the amount of computation we need to invest to find it.

There are certainly algorithms where it would be a severe issue (many RL approaches, for instance), but I'm not categorically saying that all intelligence (or approaches to such) requires a certain minimum depth. It's just that unless you already have a strong prior that easy-to-parallelize approaches will get us to AGI, the existence of Amdahl's law implies Moore slowing down is very important.

I think the brains example is somewhat misleading, for two reasons:

1: For biological anchors, people occasionally talk about brain emulation in a "simulate the interactions on a cellular level" sense (I'm not saying you do this), and this is practically the most serial task I could come up with.*

2: The brain is the inference stage of current-intelligence, not the training stage. The way we got to brains was very serial.

*(For all we know, it could be possible to parallelize every single algorithm. CS theory is weird!)

This is to some extent captured in the "headroom" point, but when I examine my own reasons for being  less worried about AI risk than the community, it's primarily about computational constraints on both the hardware and algorithmic side. 

Hardware side: We have strong reasons to believe that naive extrapolations of past (i.e. last 50 years) progress on compute will be a substantial overestimate of future progress. In particular, we face Dennard scaling failing, Moore's Law certainly slowing down and likely failing, and Amdahl's Law making both of these worse by limiting returns to parallelism (which in a world with no Dennard or Moore is effectively what increased economic resources gets you). It's also easy to implicitly be very optimistic here while forecasting e.g. bio anchors timelines, because something that looks moderate like "Moore's Law, but 50% slower" is an exponentially more optimistic assumption than what are at least plausible outcomes like "Moore's Law completely breaks". This also makes forecasts very sensitive to technical questions about computer architecture.

Algorithmic side (warning, this is more speculative): Many problems in computer science that can be precisely stated (giving us the benefit of quantifying "how much" progress there is per year) go through periods of rapid advances, where we can say quantitatively that we have moved e.g. 30% closer to the perfect matrix multiplication algorithm, or even found exponentially faster algorithms for breaking all encryption. But we have sources of belief that this progress "cannot" get us to a certain point, whether this is believing no SAT solver will ever run in 1.999^n time or even specific barrier results on all currently known ways to multiply matrices. Because of these results and beliefs, current progress (exciting as it is) causes very little updating among experts in the field that we are close to fundamental breakthroughs. The lack of these barrier results in AI could well be due to the lack of precise formulations of the problem, rather than the actual lack of these barriers.