This is a special post for quick takes by Will Aldred. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Looking back on leaving academia for EA-aligned research, here are two things I'm grateful for:
Being allowed to say why I believe something.
Being allowed to hold contradictory beliefs (i.e., think probabilistically).
In EA research, I can write: 'Mortimer Snodgrass, last author on the Godchilla paper (Gopher et al., 2021), told me "[x, y, z]".'
In academia, I had to find a previous paper to cite for any claim I made in my paper, even if I believed the claim because I heard it elsewhere. (Or, rather, I did the aforementioned for my supervisor's papers - I used to be the research assistant who cherry picked citations off Google Scholar.)
In EA research, I can write, 'I estimate that the model was produced in May 2021 (90% confidence interval: March–July 2021)', or, 'I'm about 70% confident in this claim', and even, 'This paper is more likely than not to contain an important error.'
In academia, I had to argue for a position, without conceding any ground. I had to be all-in on whatever I was claiming; I couldn't give evidence and considerations for and against. (If I did raise a counterargument, it would be as setup for a counter-counterargument.)
That's it. No further point to be made. I'm just grateful for my epistemic freedom nowadays.
One of my CERI fellows asked me to elaborate on a claim I made that was along the lines of,* "If AI timelines are shorter, then this makes (direct) nuclear risk work less important because the time during which nuclear weapons can wipe us out is shorter."
There's a general point here, I think, which isn't limited to nuclear risk. Namely, AI timelines being shorter not only makes AI risk more important, but makes everything else less important. Because the time during which the other thing (whether that be an asteroid, engineered pandemic, nuclear war, nanotech-caused grey goo scenario, etc.) matters as a human-triggered x-risk is shortened.
To give the nuclear risk example:^
If TAI is 50 years away, and per-year risk of nuclear conflict is 0.5%, then risk of nuclear conflict before TAI is 1-(0.995^50) = 22%
If TAI is 15 years away, and per-year risk of nuclear conflict is 0.5%, then risk of nuclear conflict before TAI is 1-(0.995^15) = 7%
This does rely on the assumption that we'll be playing a different ball game after TAI/AGI/HLMI arrives (if not, then there's no particular reason to view TAI or similar as a cut-off point), but to me this different ball game assumption seems fair (see, e.g., Muehlhauser, 2019).
*My background thinking behind my claim here has been inspired by conversations with Michael Aird, though I'm not certain he'd agree with everything I've written in this shortform.
^A couple of not-that-important caveats:
"Before TAI" refers to the default arrival time of TAI if nuclear conflict does not happen.
The simple calculations I've performed assume mutual independence between nuclear-risk-in-given-year-x and nuclear-risk-in-given-year-y.
*My background thinking behind my claim here has been inspired by conversations with Michael Aird, though I'm not certain he'd agree with everything I've written in this shortform.
From a skim, I agree with everything in this shortform and think it's important, except maybe "to me this different ball game assumption seems fair".
I'd say this "different ball game" assumption seems at least 50% likely to be at least roughly true. But - at least given the current limits of my knowledge and thinking - it doesn't seem 99% likely to be almost entirely true, and I think the chance it may be somewhat or very untrue should factor into our cause prioritisation & our strategies. (But maybe that's what you meant by "seems fair".)
I expand on this in this somewhat longwinded comment. I'll copy that in a reply here for convenience. (See the link for Ajeya Cotra replying and me replying to that.)
My comment on Ajeya Cotra's AMA, from Feb 2021 (so probably I'd write it differently today):
"[I'm not sure if you've thought about the following sort of question much. Also, I haven't properly read your report - let me know if this is covered in there.]
I'm interested in a question along the lines of "Do you think some work done before TAI is developed matters in a predictable way - i.e., better than 0 value in expectation - for its effects on the post-TAI world, in ways that don't just flow through how the work affects the pre-TAI world or how the TAI transition itself plays out? If so, to what extent? And what sort of work?"
An example to illustrate: "Let's say TAI is developed in 2050, and the 'TAI transition' is basically 'done' by 2060. Could some work to improve institutional decision-making be useful in terms of how it affects what happens from 2060 onwards, and not just via reducing x-risk (or reducing suffering etc.) before 2060 and improving how the TAI transition goes?"
But I'm not sure it's obvious what I mean by the above, so here's my attempt to explain:
The question of when TAI will be developed[1] is clearly very important to a whole bunch of prioritisation questions. One reason is that TAI - and probably the systems leading up to it - will very substantially change how many aspects of how society works. Specifically, Open Phil has defined TAI as "AI that precipitates a transition comparable to (or more significant than) the agricultural or industrial revolution" (and Muehlhauser has provided some more detail on what is meant by that).
But I think some EAs implicitly assume something stronger, along the lines of:
The expected moral value of actions we take now is entirely based on those actions' effects on what happens before TAI is developed and those actions' effects on the development, deployment, etc. of TAI. That is, the expected value of the actions we take now is not partly based on how the actions affect aspects of the post-TAI world in ways unrelated to how TAI is developed, deployed, etc. This is either because we just can't at all predict those effects or because those effects wouldn't be important; the world will just be very shaken up and perhaps unrecognisable, and any effects of pre-TAI actions will be washed out unless they affect how the TAI transition occurs.
E.g., things we do now to improve institutional decision-making or reduce risks of war can matter inasmuch as they reduce risks before TAI and reduce risks from TAI (and maybe also reduce actual harms, increase benefits, etc.). But they'll have no even-slightly-predictable or substantial effect on decision-making or risks of war in the post-TAI world.
But I don't think that necessarily follows from how TAI is defined. E.g., various countries, religious, ideologies, political systems, technologies, etc., existed both before the Industrial Revolution and for decades/centuries afterwards. And it seems like some pre-Industrial-Revolution actions - e.g. people who pushed for democracy or the abolition of slavery - had effects on the post-Industrial-Revolution world that were probably predictably positive in advance and that weren't just about affecting how the Industrial Revolution itself occurred.
(Though it may have still been extremely useful for people taking those actions to know that, when, where, and how the IR would occur, e.g. because then they could push for democracy and abolition in the countries that were about to become much more influential and powerful.)
So I'm tentatively inclined to think that some EAs are assuming that short timelines pushes against certain types of work more than it really does, and that certain (often "broad") interventions could be in expectation useful for influencing the post-TAI world in a relatively "continuous" way. In other words, I'm inclined to thinks there might be less of an extremely abrupt "break" than some people seem to think, even if TAI occurs. (Though it'd still be quite extreme by many standards, just as the Industrial Revolution was.)
[1] Here I'm assuming TAI will be developed, which is questionable, though it seems to me pretty much guaranteed unless some existential catastrophe occurs beforehand."
There have been times in the past (e.g., here) when I've wished there were a reaction feature, and I agree with the LessWrong post's thesis that a reaction feature would positively shape forum culture.
Hi Will, we're playing with some designs for reactions now. One question we have is whether to introduce reactions at the comment level or the post level. Do you have any gut takes on that?
Some pros and cons to introducing reactions at the post level:
Pros
It’d be nice to see positive reactions to your post from people you respect.
Heightened sense of community(?)
Cons
This’d probably make the EA Forum look less serious.
Some of the epistemic status reactions (from the LessWrong post) only really make sense at the comment level. For example, “Too Harsh” and “Missed the Point”.
I’m guessing this wouldn’t be too hard to fix, though.
It’d seem inconsistent if reactions appear at the post level, whereas agreement karma only exists at the comment level?
Inconsistent, that is, if one views karma as the “core thing” and agreement karma and reactions as additional features. (It’s not inconsistent if one views reactions as a core thing, alongside karma and above agreement karma.)
Having thought about this for five minutes or so, I think that the EA Forum looking less serious is the most important of the above considerations. Thus, my current take is that I’m in favor of reactions being introduced only at the comment level.
Also, zooming out to the meta level: is there a channel for giving feedback and suggestions on Forum design/features? I have some other hot takes that I’d be happy to share.
Interesting, thanks for your takes. One of the pros that we've been most excited about is sharing positive feedback beyond karma back with authors (some combination of your pros). The "serious" culture is super valuable, but also has the effect of scaring people away from posting their ideas, so we're thinking about what the right balance is.
Anyway, thanks for your takes! We'll probably post some ideas in the next week for more feedback.
Yeah, to clarify why I think some seriousness is important: for a number of people and orgs, this forum is the place they publish their research. Some fraction of this research will be cited outside of the EA Forum, and my guess is that non-EAs may view this research as less credible if there are, for example, smiley face reaccs alongside the title.
Nonetheless, I now think I'm leaning toward post-level reactions. Your point about sharing positive feedback back with authors is salient, in my view, and I also expect that there are viable workarounds to my seriousness objection. For instance, having epistemic status reacts (but not face emoji reacts) at the post level might get the best of both—feedback and seriousness—worlds.
(Of course, I'm just one dude with ~zero UI experience, so feel free to weight my take accordingly.)
The EA funding pool is large, but not infinite. This statement is nothing to write home about, but I've noticed quite a few EAs I talk to view longtermist/x-risk EA funding as effectively infinite, the notion being that we're severely bottlenecked by good funding opportunities.
I think this might be erroneous.
Here are some areas that could plausibly absorb all EA funding, right now:
Biorisk
Better sequencing
Better surveillance
Developing and deploying PPE
Large-scale philanthropic response to a pandemic
AI risk
Policy spending (especially in the US)
AI chips
either scaling up chip production, or buying up top-of-the-range chips
Backing the lab(s) that we might want to get to TAI/AGI/HLMI/PASTA first
(Note: I'm definitely not saying we should fund these things, but I am pointing out that there are large funding opportunities out there which potentially meet the funding bar. For what it's worth, my true thinking is something closer to: "We should reserve most of our funding for shaping TAI come crunch time, and/or once we have better strategic clarity."
Note also: Perhaps some, or all, of these don't actually work, and perhaps there are many more examples I'm missing - I only spent ~3 mins brainstorming the above. I'm also pretty sure this wasn't a totally original brainstorm, and that I was remembering these examples having read something on a similar topic somewhere, probably here on the Forum, though I can't recall which post it was.)
[This comment is no longer endorsed by its author]Reply
Hmm, it feels unclear to me what you're claiming here. In particular, I'm not sure which of the following is your claim:
"Right now all money committed to EA could be spent on things that we currently (should) think are at least slightly net positive in expectation. (Even if we maybe shouldn't spend on those things yet, since maybe we should wait for even better opportunities.)"
"Right now all money committed to EA could be spent on things that might be net positive in expectation. (But there aren't enough identified opportunities that we currently think are net positive to absorb all current EA money. Some of the things currently look net negative but with high uncertainty, and we need to do further research or wait till things naturally become closer and clearer to find out which are net positive. We also need to find more opportunities.)"
1 is a stronger and more interesting claim than 2. But you don't seem to make it clear which one you're saying.
If 2 is true, then we still are "severely bottlenecked by good funding opportunities" + by strategic clarity. So it might be that the people you're talking to are already thinking 2, rather than that EA funding is effectively infinite?
To be clear, I do think 2 is importantly different from "we have effectively infinite money", in particular in that it pushes in favor of not spending on extremely slightly net positive funding opportunities now since we want to save money for when we've learned more about which of the known maybe-good huge funding opportunties are good.* So if there are people acting and thinking as though we have effectively infinite money, I do think they should get ~this message. But I think your shortform could maybe benefit from distinguishing 1 and 2.
(Also, a nit-picky point: I'd suggest avoiding phrasing like "could plausibly absorb all EA funding" without a word like "productively", since of course there are things that can literally just absorb our funding - literally just spending is easy.)
*E.g., personally I think trying to spend >$1b in 2023 on each of the AI things you mentioned would probably require spending on some net negative in expectation things, but I also think that we should keep those ideas in mind for future and spend a bit slower on other things for that reason.
I'm also pretty sure this wasn't a totally original brainstorm, and that I was remembering these examples having read something on a similar topic somewhere, probably here on the Forum, though I can't recall which post it was
I'm also pretty sure this wasn't a totally original brainstorm, and that I was remembering these examples having read something on a similar topic somewhere, probably here on the Forum, though I can't recall which post it was.
Maybe it was some combination of the posts with the megaprojects tag?
Looking back on leaving academia for EA-aligned research, here are two things I'm grateful for:
In EA research, I can write: 'Mortimer Snodgrass, last author on the Godchilla paper (Gopher et al., 2021), told me "[x, y, z]".'
In academia, I had to find a previous paper to cite for any claim I made in my paper, even if I believed the claim because I heard it elsewhere. (Or, rather, I did the aforementioned for my supervisor's papers - I used to be the research assistant who cherry picked citations off Google Scholar.)
In EA research, I can write, 'I estimate that the model was produced in May 2021 (90% confidence interval: March–July 2021)', or, 'I'm about 70% confident in this claim', and even, 'This paper is more likely than not to contain an important error.'
In academia, I had to argue for a position, without conceding any ground. I had to be all-in on whatever I was claiming; I couldn't give evidence and considerations for and against. (If I did raise a counterargument, it would be as setup for a counter-counterargument.)
That's it. No further point to be made. I'm just grateful for my epistemic freedom nowadays.
TAI makes everything else less important.
One of my CERI fellows asked me to elaborate on a claim I made that was along the lines of,* "If AI timelines are shorter, then this makes (direct) nuclear risk work less important because the time during which nuclear weapons can wipe us out is shorter."
There's a general point here, I think, which isn't limited to nuclear risk. Namely, AI timelines being shorter not only makes AI risk more important, but makes everything else less important. Because the time during which the other thing (whether that be an asteroid, engineered pandemic, nuclear war, nanotech-caused grey goo scenario, etc.) matters as a human-triggered x-risk is shortened.
To give the nuclear risk example:^
If TAI is 50 years away, and per-year risk of nuclear conflict is 0.5%, then risk of nuclear conflict before TAI is 1-(0.995^50) = 22%
If TAI is 15 years away, and per-year risk of nuclear conflict is 0.5%, then risk of nuclear conflict before TAI is 1-(0.995^15) = 7%
This does rely on the assumption that we'll be playing a different ball game after TAI/AGI/HLMI arrives (if not, then there's no particular reason to view TAI or similar as a cut-off point), but to me this different ball game assumption seems fair (see, e.g., Muehlhauser, 2019).
*My background thinking behind my claim here has been inspired by conversations with Michael Aird, though I'm not certain he'd agree with everything I've written in this shortform.
^A couple of not-that-important caveats:
From a skim, I agree with everything in this shortform and think it's important, except maybe "to me this different ball game assumption seems fair".
I'd say this "different ball game" assumption seems at least 50% likely to be at least roughly true. But - at least given the current limits of my knowledge and thinking - it doesn't seem 99% likely to be almost entirely true, and I think the chance it may be somewhat or very untrue should factor into our cause prioritisation & our strategies. (But maybe that's what you meant by "seems fair".)
I expand on this in this somewhat longwinded comment. I'll copy that in a reply here for convenience. (See the link for Ajeya Cotra replying and me replying to that.)
My comment on Ajeya Cotra's AMA, from Feb 2021 (so probably I'd write it differently today):
"[I'm not sure if you've thought about the following sort of question much. Also, I haven't properly read your report - let me know if this is covered in there.]
I'm interested in a question along the lines of "Do you think some work done before TAI is developed matters in a predictable way - i.e., better than 0 value in expectation - for its effects on the post-TAI world, in ways that don't just flow through how the work affects the pre-TAI world or how the TAI transition itself plays out? If so, to what extent? And what sort of work?"
An example to illustrate: "Let's say TAI is developed in 2050, and the 'TAI transition' is basically 'done' by 2060. Could some work to improve institutional decision-making be useful in terms of how it affects what happens from 2060 onwards, and not just via reducing x-risk (or reducing suffering etc.) before 2060 and improving how the TAI transition goes?"
But I'm not sure it's obvious what I mean by the above, so here's my attempt to explain:
The question of when TAI will be developed[1] is clearly very important to a whole bunch of prioritisation questions. One reason is that TAI - and probably the systems leading up to it - will very substantially change how many aspects of how society works. Specifically, Open Phil has defined TAI as "AI that precipitates a transition comparable to (or more significant than) the agricultural or industrial revolution" (and Muehlhauser has provided some more detail on what is meant by that).
But I think some EAs implicitly assume something stronger, along the lines of:
But I don't think that necessarily follows from how TAI is defined. E.g., various countries, religious, ideologies, political systems, technologies, etc., existed both before the Industrial Revolution and for decades/centuries afterwards. And it seems like some pre-Industrial-Revolution actions - e.g. people who pushed for democracy or the abolition of slavery - had effects on the post-Industrial-Revolution world that were probably predictably positive in advance and that weren't just about affecting how the Industrial Revolution itself occurred.
(Though it may have still been extremely useful for people taking those actions to know that, when, where, and how the IR would occur, e.g. because then they could push for democracy and abolition in the countries that were about to become much more influential and powerful.)
So I'm tentatively inclined to think that some EAs are assuming that short timelines pushes against certain types of work more than it really does, and that certain (often "broad") interventions could be in expectation useful for influencing the post-TAI world in a relatively "continuous" way. In other words, I'm inclined to thinks there might be less of an extremely abrupt "break" than some people seem to think, even if TAI occurs. (Though it'd still be quite extreme by many standards, just as the Industrial Revolution was.)
[1] Here I'm assuming TAI will be developed, which is questionable, though it seems to me pretty much guaranteed unless some existential catastrophe occurs beforehand."
I'd heart react if this forum introduced reactions.[1]
There have been times in the past (e.g., here) when I've wished there were a reaction feature, and I agree with the LessWrong post's thesis that a reaction feature would positively shape forum culture.
Hi Will, we're playing with some designs for reactions now. One question we have is whether to introduce reactions at the comment level or the post level. Do you have any gut takes on that?
That’s great news!
Some pros and cons to introducing reactions at the post level:
Having thought about this for five minutes or so, I think that the EA Forum looking less serious is the most important of the above considerations. Thus, my current take is that I’m in favor of reactions being introduced only at the comment level.
Also, zooming out to the meta level: is there a channel for giving feedback and suggestions on Forum design/features? I have some other hot takes that I’d be happy to share.
Interesting, thanks for your takes. One of the pros that we've been most excited about is sharing positive feedback beyond karma back with authors (some combination of your pros). The "serious" culture is super valuable, but also has the effect of scaring people away from posting their ideas, so we're thinking about what the right balance is.
Anyway, thanks for your takes! We'll probably post some ideas in the next week for more feedback.
You can give feature suggestions here any time.
Yeah, to clarify why I think some seriousness is important: for a number of people and orgs, this forum is the place they publish their research. Some fraction of this research will be cited outside of the EA Forum, and my guess is that non-EAs may view this research as less credible if there are, for example, smiley face reaccs alongside the title.
Nonetheless, I now think I'm leaning toward post-level reactions. Your point about sharing positive feedback back with authors is salient, in my view, and I also expect that there are viable workarounds to my seriousness objection. For instance, having epistemic status reacts (but not face emoji reacts) at the post level might get the best of both—feedback and seriousness—worlds.
(Of course, I'm just one dude with ~zero UI experience, so feel free to weight my take accordingly.)
Great, thanks!
We could spend all longtermist EA money, now.
(This is a sort-of sequel to my previous shortform.)
The EA funding pool is large, but not infinite. This statement is nothing to write home about, but I've noticed quite a few EAs I talk to view longtermist/x-risk EA funding as effectively infinite, the notion being that we're severely bottlenecked by good funding opportunities.
I think this might be erroneous.
Here are some areas that could plausibly absorb all EA funding, right now:
(Note: I'm definitely not saying we should fund these things, but I am pointing out that there are large funding opportunities out there which potentially meet the funding bar. For what it's worth, my true thinking is something closer to: "We should reserve most of our funding for shaping TAI come crunch time, and/or once we have better strategic clarity."
Note also: Perhaps some, or all, of these don't actually work, and perhaps there are many more examples I'm missing - I only spent ~3 mins brainstorming the above. I'm also pretty sure this wasn't a totally original brainstorm, and that I was remembering these examples having read something on a similar topic somewhere, probably here on the Forum, though I can't recall which post it was.)
Hmm, it feels unclear to me what you're claiming here. In particular, I'm not sure which of the following is your claim:
1 is a stronger and more interesting claim than 2. But you don't seem to make it clear which one you're saying.
If 2 is true, then we still are "severely bottlenecked by good funding opportunities" + by strategic clarity. So it might be that the people you're talking to are already thinking 2, rather than that EA funding is effectively infinite?
To be clear, I do think 2 is importantly different from "we have effectively infinite money", in particular in that it pushes in favor of not spending on extremely slightly net positive funding opportunities now since we want to save money for when we've learned more about which of the known maybe-good huge funding opportunties are good.* So if there are people acting and thinking as though we have effectively infinite money, I do think they should get ~this message. But I think your shortform could maybe benefit from distinguishing 1 and 2.
(Also, a nit-picky point: I'd suggest avoiding phrasing like "could plausibly absorb all EA funding" without a word like "productively", since of course there are things that can literally just absorb our funding - literally just spending is easy.)
*E.g., personally I think trying to spend >$1b in 2023 on each of the AI things you mentioned would probably require spending on some net negative in expectation things, but I also think that we should keep those ideas in mind for future and spend a bit slower on other things for that reason.
Perhaps thinking of this post?
Maybe it was some combination of the posts with the megaprojects tag?