Stuart Buck
Bio
I lead a small think tank dedicated to accelerating the pace of scientific advancement by improving the conditions of science funding. As well, I'm a senior advisor to the Social Science Research Council. Prior to these roles, I spent some 9 years at Arnold Ventures (formerly the Laura and John Arnold Foundation) as VP of Research.
How I can help others
Science policy, reproducibility, and philanthropy.
Posts 5
Comments49
Because the question is impossible to answer.
First, by definition, we have no actual evidence about outcomes in the long-term future--it is not as if we can run RCTs where we run Earth 1 for the next 1,000 years with one intervention and Earth 2 with a different intervention. Second, even where experts stand behind short-term treatments and swear that they can observe the outcomes happening right in front of them (everything from psychology to education to medicine), there are many cases where the experts are wrong -- even many cases where we do harm while thinking we do good (see Prasad and Cifu's book Medical Reversals).
Given the lack of evidentiary feedback as well as any solid basis for considering people to be "experts" in the first place, there is a high likelihood that anything we think benefits the long-term future might do nothing or actually make things worse.
The main way to justify long-termist work (especially on AGI) is to claim that there's a risk of everyone dying (leading to astronomically huge costs), and then claim that there's a non-zero positive probability of affecting that outcome. There will never be any evidentiary confirmation of either claim, but you can justify any grant to anyone for anything by adjusting the estimated probabilities as needed.
It's all a bit intuitive, but my heuristics were basically: Figure out the general issues that seem worth addressing; find talented people who are already trying to address those issues (perhaps in their spare time) and whose main constraint is capital; and give them more capital (e.g., time and employees) to do even better things (which they will often come up with on their own).
I've been a grantmaker (at Arnold Ventures, a $2 billion philanthropy), and I couldn't agree more. Those kinds of questions are good if the aim is to reward and positively select for people who are good at bullshitting. And I also worry about a broader paradox -- sometimes the highest impact comes from people who weren't thinking about impact, had no idea where their plans would lead, and serendipitously stumbled into something like penicillin while doing something else.
Just a note: this post could have opposite advice for people from guess culture rather than ask culture. See https://ask.metafilter.com/55153/Whats-the-middle-ground-between-FU-and-Welcome#830421
I.e., someone from ask culture might need to be warned not to bother people so much. Someone from guess culture might need to be told that it is ok to reach out to people once in a while.
"I think all of these considerations in-aggregate make me worried that a lot of current work in AI Alignment field-building and EA-community building is net-negative for the world, and that a lot of my work over the past few years has been bad for the world"
This admirably honest statement deserves more emphasis. As we know from medicine and international development and anywhere that does RCTs, it is really, really hard -- even when the results of your actions are right in front of you -- to know whether you have helped someone or harmed them. There are just too many confounding factors, selection bias, etc.
The long-termist AGI stuff has always struck me as even worse off in this respect. How is anyone supposed to know that the actions they take today will have a beneficial impact on the world decades from now, rather than making things worse? And given the premises of AGI alignment, making things worse would be utterly catastrophic for humanity.
"But a true AGI could not only transform the world, it could also transform itself."
Is there a good argument for this point somewhere? It doesn't seem obvious at all. We are generally intelligent ourselves, and yet existed for hundreds of thousands of years before we even discovered that there are neurons, synapses, etc., and we are absolutely nowhere near the ability to rewire our neurons and glial cells so as to produce ever-increasing intelligence. So too, if AGI ever exists, it might be at an emergent level that has no idea it is made out of computer code, let alone knows how to rewrite its own code.
I generally agree, but I think that we are nowhere near being able to say, "The risk of future climate catastrophe was previously 29.5 percent, but thanks to my organization's work, that risk has been reduced to 29.4 percent, thus justifying the money spent." The whole idea of making grants on such a slender basis of unprovable speculation is radically different from the traditional EA approach of demanding multiple RCTs. Might be a great idea, but still a totally different thing. Shouldn't even be mentioned in the same breath.