Updated terminology on October 2, 2022.

Many thanks for feedback and insight from Kelly Anthis, Tobias Baumann, Jan Brauner, Max Carpendale, Sasha Cooper, Sandro Del Rivo, Michael Dello-Iacovo, Michael Dickens, Anthony DiGiovanni, Marius Hobbhahn, Ali Ladak, Simon Knutsson, Greg Lewis, Kelly McNamara, John Mori, Thomas Moynihan, Caleb Ontiveros, Sean Richardson, Zachary Rudolph, Manny Rutinel, Stefan Schubert, Michael St. Jules, Nell Watson, Peter Wildeford, and Miranda Zhang. This essay is in part an early draft of an upcoming book chapter on the topic, and I will add the citation here when it is available.

Our lives are not our own. From womb to tomb, we are bound to others, past and present. And by each crime and every kindness, we birth our future. ⸻ Cloud Atlas (2012)

Summary

The prioritization of extinction risk reduction depends on an assumption that the expected value (EV)^[1] of human survival and interstellar colonization is highly positive. In the feather-ruffling spirit of EA Criticism and Red Teaming, this essay lays out many arguments for a positive EV and a negative EV. This matters because, insofar as the EV is lower than we previously believed, we should shift some longtermist resources away from the current focus on extinction risk reduction. Extinction risks are the most extreme category of population risks, which are risks to the number of individuals in the long-term future. We could shift resources towards the other type of long-term risk, quality risks, which are risks to the moral value of individuals in the long-term future, such as whether they experience suffering or happiness.^[2] Promising approaches to improve the quality of the long-term future include some forms of AI safety, moral circle expansion, cooperative game theory, digital minds, and global priorities research. There may be substantial overlap with extinction risk reduction approaches, but in this case and in general, much more research is needed. I think that the effective altruism (EA) emphasis on existential risk could be replaced by a mindset of cautious longtermism:

Before humanity colonizes the universe, we must ensure that the future we would build is one worth living in.

I have spoken to many longtermist EAs about this crucial consideration, and for most of them, that was their first time explicitly considering the EV of human expansion.^[3] My sense is ...

Longtermist Theory