Great post. I basically agree, but in a spirit of devil's advocating, I will say: when I turn my mind to agent foundations thinking, I often find myself skirting queasily close to concepts which feel also capabilities-relevant (to the extent that I have avoided publicly airing several ideas for over a year).

I don't know if that's just me, but it does seem that some agent foundations content from the past has also had bearing on AI capabilities - especially if we include decision theory stuff, dynamic programming and RL, search, planning etc. which it's arguably artificial to exclude. How would you ex ante distinguish e.g. work which explores properties and constraints of hypothetical planning routines from work which informs creation of more effective planning routines? This sort of thinking seems relevant.

I wrote a little here about unpluggability (and crossposted on LessWrong/AF)

Thank you for sharing this! Especially the points about relevant maps and Meta/FAIR/LeCun.

I was recently approached by the UK FCDO as a technical expert in AI with perspective on x-risk. We had what I think were very productive conversations, with an interesting convergence of my framings and the ones you've shared here - that's encouraging! If I find time I'm hoping to write up some of my insights soon.

I've given a little thought to this hidden qualia hypothesis but it remains very confusing for me.

To what extent should we expect to be able to tractably and knowably affect such hidden qualia?

This is beautiful and important Tyler, thank you for sharing.

I've seen a few people burn out (and come close myself), and I have made a point of gently socially making and reinforcing this sort of point (far less eloquently) myself, in various contexts. 

I have a lot of thoughts about this subject.

One thing I embrace always is silliness and (often self-deprecating) humour, which are useful antidotes to stress for a lot of people. Incidentally, your tweet thread rendition of the Eqyptian spell includes

I am light heading for light. Even in the dark, a fire bums in the distance.

(emphasis mine) which I enjoyed. A case of bad keming reified?

A few friends and acquaintances have recently been working on something they're calling Shard Theory, which considers the various parts of a human's motivation system and their interactions. They're interested for other reasons, but I was reminded here. See also Kaj Sotala's Multiagent Models of Mind which is more explicitly about how to be a human.

As a firm descriptive (but undecidedly prescriptive) transhumanist, I think your piece here also touches on something we will likely one day (maybe soon?) have to grapple with, which is the fundamental relationship between (moral) agency and moral patienthood. As it happens, modern humans are quite conclusively both, by most lights, but it doesn't look like this is a law of nature. Indeed there are likely many deserving moral patients today who are not much by way of agents. And we may bring into being agents which are not especially moral-patienty. (Further, something sufficiently agenty might render humans themselves ourselves to the status of 'not much by way of agents'.)

Seconded/thirded on Human Compatible being near that frontier. I did find its ending 'overly optimistic' in the sense of framing it like 'but lo, there is a solution!' while other similar resources like Superintelligence and especially The Alignment Problem seem more nuanced in presenting uncertain proposals for paths forward not as oven-ready but preliminary and speculative.

I think it's a staircase? Maybe like climbing upwards to more good stuff. Plus some cool circles to make it logo ish.

I'm intrigued by this thread. I don't have an informed opinion on the particular aesthetic or choice of quiz questions, but I note some superficial similarities to Coursera, Khan Academy, and TED-Ed, which are aimed at mainly professional age adults, students of all ages, and youth/students (without excluding adults) respectively.

Fun/cute/cartoon aesthetics do seem to abound these days in all sorts of places, not just for kids.

My uninformed opinion is that I don't see why it should put off teenagers (talented or otherwise) in particular, but I weakly agree that if something is explicitly pitched at teenagers, that might be offputting!

It looks like I got at least one downvote on this comment. Should I be providing tips of this kind in a different way?