Updated terminology on October 2, 2022.
Many thanks for feedback and insight from Kelly Anthis, Tobias Baumann, Jan Brauner, Max Carpendale, Sasha Cooper, Sandro Del Rivo, Michael Dello-Iacovo, Michael Dickens, Anthony DiGiovanni, Marius Hobbhahn, Ali Ladak, Simon Knutsson, Greg Lewis, Kelly McNamara, John Mori, Thomas Moynihan, Caleb Ontiveros, Sean Richardson, Zachary Rudolph, Manny Rutinel, Stefan Schubert, Michael St. Jules, Nell Watson, Peter Wildeford, and Miranda Zhang. This essay is in part an early draft of an upcoming book chapter on the topic, and I will add the citation here when it is available.
Our lives are not our own. From womb to tomb, we are bound to others, past and present. And by each crime and every kindness, we birth our future. ⸻ Cloud Atlas (2012)
Summary
The prioritization of extinction risk reduction depends on an assumption that the expected value (EV)[1] of human survival and interstellar colonization is highly positive. In the feather-ruffling spirit of EA Criticism and Red Teaming, this essay lays out many arguments for a positive EV and a negative EV. This matters because, insofar as the EV is lower than we previously believed, we should shift some longtermist resources away from the current focus on extinction risk reduction. Extinction risks are the most extreme category of population risks, which are risks to the number of individuals in the long-term future. We could shift resources towards the other type of long-term risk, quality risks, which are risks to the moral value of individuals in the long-term future, such as whether they experience suffering or happiness.[2] Promising approaches to improve the quality of the long-term future include some forms of AI safety, moral circle expansion, cooperative game theory, digital minds, and global priorities research. There may be substantial overlap with extinction risk reduction approaches, but in this case and in general, much more research is needed. I think that the effective altruism (EA) emphasis on existential risk could be replaced by a mindset of cautious longtermism:
Before humanity colonizes the universe, we must ensure that the future we would build is one worth living in.
I have spoken to many longtermist EAs about this crucial consideration, and for most of them, that was their first time explicitly considering the EV of human expansion.[3] My sense is that many more are considering it now, and the community is growing more skeptical of highly positive EV as the correct estimate. I’m eager to hear more people’s thoughts on the all-things-considered estimate of EV, and I discuss the limited work done on this topic to date in the “Related Work” section.
In the following table, I lay out the object-level arguments on the EV of human expansion, and the rest of the essay details meta-considerations (e.g., option value). The table also includes the strongest supporting arguments that increase the evidential weight of their corresponding argument and the strongest counterarguments that reduce the weight. The arguments are not mutually exclusive and are merely intended as broad categories that reflect the most common and compelling arguments for at least some people (not necessarily me) on this topic. For example, Historical Progress and Value Through Intent have been intertwined insofar as humans intentionally create progress, so users of this table should be mindful that they do not overcount (e.g., double count) the same evidence. I handle this in my own thinking by splitting an overlapping piece of evidence among its categories in proportion to a rough sense of fit in those categories.[4]
In the associated spreadsheet, I list my own subjective evidential weight scores where positive numbers indicate evidence for +EV and negative numbers indicate evidence for -EV. It is helpful to think through these arguments with different assignment and aggregation methods, such as linear or logarithmic scaling. With different methodologies to aggregate my own estimates or those of others, the total estimate is highly negative around 30% of the time, weakly negative 40%, and weakly positive 30%. It is almost never highly positive. I encourage people to make their own estimates, and while I think such quantifications are usually better than intuitive gestalts, all such estimates should be taken with golf balls of salt.[5]
This is an atypical structure for an argumentative essay—laying out all the arguments, for and against, instead of laying out arguments for my position and rebutting the objections—but I think that we should detach argumentation from evaluation. I’m not aiming for maximum persuasiveness. Indeed, the thrust of my critique is that EAs have failed to consider these arguments in such a systematic way, either neglecting the assumption entirely or selecting only a handful of the multitude of evidence and reason we have available. Overall, my current thinking (primarily an average of several aggregations of quantified estimates and Aumann updating on others’ views) is that the EV of human expansion is not highly positive. For this and other reasons, I prioritize improving the quality of the long-term future rather than increasing its expected population.
Arguments on the Expected Value (EV) of Human Expansion
Argument Name | Description |
Arguments for Positive Expected Value (EV) | |
Historical Progress | Humanity has achieved great progress in adding value and reducing disvalue, especially since the Enlightenment, such as through declines in violence, oppression, disease, and poverty. In particular, explicit human values seem to have progressed alongside human behavior, which may more robustly extend into the long-term future. Many scholars have written persuasively on this evidence, most famously Pinker (2012; 2018).
|
Value Through Intent | As technology increases, it arguably seems that (i) humans exert more of their intent on the universe, and (ii) humans tend to want good more than bad.
|
Value Through Evolution | Evolution (e.g., selection of genetic material over generations) selects for some forms of value and good moral attitudes, at least for oneself. Altruism and self-sacrifice can be selected for (e.g., in soldier ants), especially insofar as altruists care more about future generations. These forces may apply to the evolution of post-humans, AGIs, or minds created by an unaligned/rogue AGI. Christiano (2013) argues that longtermist values will be selected for over time, though it is unclear how this applies to non-temporal sorts of altruism.
|
Convergence of Patiency and Agency | Moral patients (i.e., beings who can have positive or negative value) may tend to be agents able to protect their own interests (e.g., to exit a situation when it is disvaluable). In other words, if more patients are agents, that's reason for optimism because such beings can use their power as agents to protect their moral interests as patients. A society with many such beings may be the most likely type of long-term society for various reasons, such as if patiency turns out to not be very useful for enacting the will of agents with power: Insofar as digital minds are the most numerous minds in the long-term future, builders may be able to opt to not make minds doing potentially suffering-inducing activities have the capacity to suffer, though this depends on the usefulness of suffering and only applies to certain endeavors. |
Reasoned Cooperation | Agents tend to selfishly benefit from working together, such as in families, herds, villages, city-states, nations, and international trade. Such cooperation may protect the interests of future beings. For example, we could expect similar cooperation to evolve on alien worlds or in any evolutionary forces behind digital mind development.
|
Discoverable Moral Reality | If there are stance-independent moral facts (e.g., divine moral truth), then future beings may discover and implement them.
|
Arguments for Negative Expected Value (EV) | |
Historical Harms | Humanity has a very bad track record of harming other humans as well as domestic and wild animals. The empirical evidence for disvalue seems clearest to people who have worked on human and animal rights issues because of salient firsthand experience with cruel and callous humans can be, particularly the unsettling “seriousness of suffering” (see the disturbing examples in Tomasik 2006 for an introduction). This is a topic we are very tempted to ignore, downplay, or rationalize (see Cohen 2001 and “Biases” below). The largest sources of disvalue today are factory farming and wild animal suffering (Anthis 2016b).
|
Disvalue Through Intent | Many human intentions cause harm to others, such as desires for power, status, and novelty. There are many plausible human interstellar endeavors that involve extensive disvalue in ways that may not be avoided with mere technological advancement (as, arguably, factory farming of animals will be avoided), such as “recreation (e.g. safaris, war games), a labor force (e.g. colonists to distant parts of the galaxy, construction workers), scientific experiments, threats, (e.g. threatening to create and torture beings that a rival cares about), revenge, justice, religion, or even pure sadism” (Anthis 2018b).
|
Disvalue Through Evolution | Evolution tends to produce more suffering than happiness, such as in wild animals.
|
Divergence of Patiency and Agency | Moral patients (i.e., beings who can have positive or negative value) may tend to not be agents able to protect their own interests (e.g., to exit a situation when it is disvaluable). This may be the most likely type of long-term society for various reasons (Anthis 2018).
|
Threats | Disvalue can be used as a threat, such as threatening to torture many simulated copies of another agent unless they hand over some of their interstellar resources.
|
Treadmills | Even people with great material resources can be very unhappy, including many of the best-off humans today, such as in the Easterlin Paradox (Plant 2022).
|
Arguments that May Increase or Decrease Expected Value (EV) | |
Conceptual Utility Asymmetry | A unit of disvalue (e.g., suffering) may be larger in absolute value from a unit of value (e.g., happiness). This could be an axiological asymmetry between some natural units of disvalue and value, or empirical (see below).
|
Empirical Utility Asymmetry | A unit of disvalue (e.g., suffering) may be larger in absolute value from a unit of value (e.g., happiness), or vice versa. This could be an axiological asymmetry (see above) or an empirical asymmetry, such as between per-joule units of disvalue and value or between dolorium and hedonium. As described in Anthis and Paez (2021), when we imagine the largest values and disvalues (e.g., How many days of intense pleasure would you trade for intense pain?), the disvalues tend to seem larger. This argument overlaps with most other arguments in this table, so users should be cautious about overcounting the same evidence. |
Complexity Asymmetry | Disvalue may be simpler and thus easier to produce and more common than value. This is a variation of the Anna Karenina principle that failure tends to come from any of a number of factors, which was posed at least as early as Aristotle’s Nichomachean Ethics. Value, on the other hand, may be more complex, a view favored by some in AI safety, such as Yudkowsky (2007). The opposite argument may obtain, though I have never heard anyone believe that claim. |
Procreation Asymmetry | Bringing value-positive people into existence may be less valuable than adding value to existing people, but bringing negative-value people into existence may not be as different from adding disvalue to existing people—or vice versa.
|
EV of the Counterfactual | If humans do not expand, perhaps because we die off or stay on Earth, what will the EV of the universe be? This counterfactual EV can include wild animals on Earth (including those who could evolve a human-like society after many years of a humanless Earth if humans die off), alien civilizations (who may be very different from humans, such as evolving more like insects or solitary predators), value or disvalue in the universe as we know it (e.g., stars being born and dying, fundamental physics in which particles are attracted and repelled by each other, Boltzmann brains), parallel universes (whom we may otherwise affect, for better or worse, through acausal interactions or as-yet-undiscovered causal mechanisms), and simulators (if we live in a simulation). Human expansion may lead to increases or decreases in the EV of these groups. Depending on what sort of expansion we’re considering, such as if we curtail the +EV or -EV expansion of alien civilizations or if we attack or rescue them, this counterfactual may also include unaligned AI systems that kill humans or prevent our expansion but expand themselves, such as by paperclipping the universe (which may involve many paperclipping drones and von Neumann probes). The EV of aligned versus unaligned AI systems has been discussed in Tomasik (2015) and Gloor (2018) from a total-suffering perspective, and it remains extremely unclear. |
The Nature of Digital Minds, People, and Sentience | While there are relatively clear advantages to digital minds over biological minds, such as the ability to self-modify, copy, and travel long distances, it is much less clear what life for digital minds would be like. For example, we do not know how much protection digital sentience will have over their own experiences (e.g., cryptographic security), how useful it will be to have many small minds versus few large minds, and how useful nesting of minds within each other will be. There are also many normative questions regarding the value of these different minds, such as group entities where the subunits are more distinct than subunits of a biological brain (e.g., What if a China brain were implemented in which there were tiny humans inside of each neuron in a normal human brain, passing around neurotransmitters?). Digital minds may also make up value-optimal structures (e.g., dolorium and hedonium). See “Scaling of Value and Disvalue” below. This argument overlaps with most other arguments in this table, so users should be cautious about overcounting the same evidence. |
Life Despite Suffering | Even in dire scenarios, many humans report a preference to live or have lived over to die or to have never been born. This may suggest underappreciated value in even apparently disvaluable lives (e.g., aspects not covered in current moral frameworks) or, as with many psychological arguments, it can be viewed as a bias of overemphasizing value in our evaluations (e.g., just world bias, fear of death). See “Biases” below. |
The Nature of Value Refinement | Many plausible trajectories of the long-term future involve some sort of value refinement, such as coherent extrapolated volition (Yudkowsky 2004), indirect normativity (Bostrom 2014), and long reflection (Greaves and MacAskill 2017). The effect of such processes on values depends on a range of questions such as: Whose values are refined? How important are value inputs at the beginning of refinement (i.e., to what extent are they locked in)? And what sort of moral considerations (e.g., thought experiments) has humanity not yet considered but may consider in such processes? Insofar as one believes that AGI will be instrumental in humanity’s future, even an AGI that is aligned in some way may not be good. It depends on what values are aligned with what aspect of the AGI. Especially concerning is that the AGI may only be aligned with human values and interests, only caring about nonhuman beings to the extent humans do, which may not be sufficient for net positive outcomes. Tomasik (2013) covers many of these arguments, e.g., “Very likely our values will be lost to entropy or Darwinian forces beyond our control. However, there's some chance that we'll create a singleton in the next few centuries that includes goal-preservation mechanisms allowing our values to be ‘locked in’ indefinitely. Even absent a singleton, as long as the vastness of space allows for distinct regions to execute on their own values without take-over by other powers, then we don't even need a singleton; we just need goal-preservation mechanisms,” as does Tomasik (2017). |
Scaling of Value and Disvalue | Sources of value and disvalue vary in magnitude, and some sources seem more likely to be value-optimized, such as dolorium as optimal suffering per unit of resource (e.g., joules of energy) or hedonium as optimal happiness. Forces such as human intent, resource accumulation, evolution, and runaway AI seem to be particularly optimizing. This consideration also depends on how values and optimization are viewed, such as what it means to optimize a layperson’s intuitionist morality. The more one cares about this sort of utilitronium or value-optimized sources (empirically or conceptually), the more such sources matter. One can also have different evaluations of dolorium and hedonium, such as whether they have a ratio of -1:1, -100:1, etc. (see Shulman 2012 and Knutsson 2017 for some discussion). This can also affect trade-offs within closer-to-zero ranges of value and disvalue. More broadly, the gradient of possible positive and negative futures could make large differences in the best approach to reducing quality risks, such as jumping from one step of EV to the one above it (e.g., futures where digital sentiences are not seen as people to futures where they are). The larger the jump between steps, the more one should prioritize even small chances of moving up a step (e.g., avoiding an existential risk). This argument overlaps with most other arguments in this table, so users should be cautious about overcounting the same evidence. |
EV of Human Expansion after Near-Extinction or Other Events | While humans may survive and colonize the cosmos along what would seem a similar trajectory to our current one, there may be major events, such as an apocalyptic near-extinction event in which the human population is decimated but recovers. It is very unclear how such events would affect the EV of human expansion. For example, post-near-extinction humans may have a newfound sense of global stewardship and camaraderie or they may have a newfound sense of resource scarcity and fear of each other. Similarly, humans after radical technology change such as life extension may have very different values, such as a resistance to changing their values the way new generations of humans do. Near-extinction events may also select for certain demographics and ideologies. |
The Zero Point of Value | Each argument for +EV and -EV depends on where one places the zero point of value. Some scenarios, such as an unaligned AI that carpets the universe with sentience that has a very limited amount of value (e.g., muzak and potatoes) and very limited amount of disvalue (e.g., boredom), may teeter on where one places the zero point between +EV and -EV. |
Related Work
Some of our successors might live lives and create worlds that, though failing to justify past suffering, would give us all, including some of those who have suffered, reasons to be glad that the Universe exists. ⸻ Derek Parfit (2017)
The field of existential risk has intellectual roots as deep as human history in notions of “apocalypse” such as the end of the Mayan calendar. Thomas Moynihan (2020) distinguishes apocalypse as having a sense to it or a justification, such as the actions of a supernatural deity, while “extinction” entails “the ending of sense” entirely. This notion of human extinction is traced back only to the Enlightenment beginning in the 1600s, and its most well-known articulation in the 21st century is under the category of existential risks (also known as x-risks), a term coined in 2002 by philosopher Nick Bostrom for risks “where an adverse outcome would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential.”
The most famous essay on existential risk is “Astronomical Waste” (Bostrom 2003), in which Bostrom argues that if humanity could colonize the Virgo supercluster, the massive concentration of galaxies that includes our own Milky Way and 47,000 of its neighbors, then we could sustain approximately 10^38 human beings, an intuitively inconceivably large number. Bostrom argues that the priority of utilitarians should be to reduce existential risk and ensure we seize this cosmic endowment, though the leap from the importance of the long-term future to existential risk reduction is contentious (e.g., Beckstead 2013b). The field of existential risk studies has risen at pace with the growth of effective altruism (EA), with a number of seminal works summarizing and advancing the field (Matheny 2007; Bostrom 2012; Beckstead 2013a; Bostrom 2013; 2014; Tegmark 2017; Russell 2019; Moynihan 2020; Ord 2020; MacAskill forthcoming).
Among existential risks, EAs have largely focused on population risks (particularly extinction risks); the term “x-risk,” which canonically refers to existential risk, is often interpreted as extinction risk (see Aird 2020a). A critical assumption underlying this focus has been that the expected value of humanity’s survival and interstellar colonization is very high.. This assumption largely goes unstated, but it was briefly acknowledged in Beckstead (2013a):
Is the expected value of the future negative? Some serious people—including Parfit (2011, Volume 2, chapter 36), Williams (2006), and Schopenhauer (1942)—have wondered whether all of the suffering and injustice in the world outweigh all of the good that we've had. I tend to think that our history has been worth it, that human well-being has increased for centuries, and that the expected value of the future is positive. But this is an open question, and stronger arguments pointing in either direction would be welcome.
Christiano (2013) asked, “Why might the future be good?” though, as I understood it, that essay did not mention the possibility of a negative future. I had also implicitly accepted the assumption of a good future until 2014, when I thought through the evidence and decided to prioritize moral circle expansion at the intersection of animal advocacy and longtermism (Anthis 2014). I brought it up on the old EA Forum in Anthis (2016a), and West (2017) detailed a version of the “Value Through Intent” argument. I also remember extensive Facebook threads around this time, though I do not have links to share. I finally wrote up my thoughts on the topic in detail in Anthis (2018b) as part of a prioritization argument for moral circle expansion over decreasing extinction risk through AI alignment, and this essay is a follow-up to and refinement of those ideas.
Later in 2018, Brauner and Grosse-Holz (2018) published an EA Forum essay arguing that the expected value of extinction risk reduction is positive. In my opinion, it failed to consider many of the arguments on the topic, as discussed in EA Forum comments and a rebuttal, also on the EA Forum, DiGiovanni (2021). There is also a chapter in MacAskill (forthcoming) covering similar ground as Brauner and Grosse-Holz, with similar arguments missing, in my opinion. Overall, these writings primarily focus on three arguments:
- the “Value Through Intent” or “will” argument, that insofar as humanity exerts its will, we tend to produce value rather than disvalue;
- the likelihood that factory farming and wild animal suffering, the largest types of suffering today, will persist into the far future; and
- axiological considerations, particularly the population ethics question of whether creating additional beings with positive welfare is morally good. This has been the main argument against increasing population from some negative utilitarians and other “suffering-focused” EAs, such as the Center on Long-Term Risk (CLR) and Center for Reducing Suffering (CRS), since Tomasik (2006).
These are three important considerations, but they cover only a small portion of the total landscape of evidence and reason that we have available for estimating the EV of human expansion. For transparency, I should flag that at least some of the authors would disagree with me about this critique of their work.
Overall, I think the arguments against a highly positive EV of human expansion have been the most important blindspot of the EA community to date. This is the only major dissenting opinion I have with the core of the EA memeplex. I would guess over 90% of longtermist EAs with whom I have raised this topic have never considered it before, despite acknowledging during our conversation that the expected value being highly positive is a crucial assumption for prioritizing extinction risk and that it is on shaky ground—if not deciding that it is altogether mistaken. (Of course, this is not meant as a representative sample of all longtermist EAs.) While examining this assumption and deciding that the far future is not highly positive would not completely overhaul longtermist EA priorities, I tentatively think that it would significantly change our focus. In particular, we should shift resources away from extinction risk and towards quality risks, including more global priorities research to better understand this and other crucial considerations. I would be eager for more discussion of this topic, and the sort of evidence I expect to most change my mind is the cooperative game theory research done by CLR, the Center on Human-Compatible AI (CHAI), and others in AI safety; the moral circle expansion and digital minds research done by Sentience Institute (SI), Future of Humanity Institute (FHI), and others in longtermism and AI safety; and all sorts of exploration of concrete scenarios similar to The Age of Em (Hanson 2016) and AI takeoff “training stories” (Hubinger 2021). I expect fewer updates from more conceptual discourse like the works cited above on the EA Forum and this essay, but I still see them as valuable contributions. See further discussion in the “Future Research on the EV of Human Expansion” subsection below.
Terminology
I separate the moral value of the long-term future into two factors: population, the number of individuals at each point in time, and quality, the moral value of each individual’s existence at each point in time. The moral value of the long-term future is thus the double sum of quality across individuals across time. Risks to the number of individuals (living sufficiently positive lives) are population risks, and risks to the quality of each individual life are quality risks.
Extinction risks are a particular sort of population risk, those that would “annihilate Earth-originating intelligent life,” though I would also include threats towards populations of non-Earth-originating and non-intelligent (and perhaps even non-living) individuals who matter morally, and I get the sense that others have also favored this more inclusive definition. Non-existential population risks could be a permanent halving of the population or a delay of one-third the universe’s remaining lifetime in humanity’s interstellar expansion, though there is no consensus on where exactly the cutoff is between existential and non-existential, though there does seem to be consensus that extinction of humans (with no creation of post-humans, such as whole brain emulations) is existential.
Quality risks are risks to the moral value of individuals who may exist in the long-term future. Existential quality risks are those that “permanently and drastically curtail its potential” moral value, such as all individuals being moved from positive to zero or positive to negative value. Non-existential quality risks may include one-tenth of the future population dropping from highly positive to barely positive quality, one-fourth of the future population dropping from barely positive to barely negative quality, and so on. Again, this may be better understood as a spectrum of existentiality, rather than two neatly separated categories, because it is unclear at what point potential is permanently and drastically curtailed. Quality risks include suffering risks (also known as_ s-risks_), “risks of events that bring about suffering in cosmically significant amounts” (Althaus and Gloor 2016; Tomasik 2011), which was noted as “weirdly sidelined” by total utilitarians in Rowe’s (2022) “Critiques of EA that I Want to Read.”
These categories are not meant to coincide with the existential risk taxonomies of Bostrom (2002) (bangs, crunches, shrieks, whimpers) or Bostrom (2013) (human extinction, permanent stagnation, flawed realization, subsequent ruination), in part because those are worded in terms of positive potential rather than an aggregation of positive and negative outcomes. However, one can reasonably view some of those categories (e.g., shrieks and failed realizations) as including some positive, zero, or negative quality trajectories because they have a failed realization of positive potential. Aird (2020b) has some useful Venn diagrams of the overlaps of some long-term risks.
The term “trajectory change” has variously been used as a category that, from my understanding, includes the mitigation or exacerbation of all of the risks above, such as Beckstead’s (2013a) definition of trajectory changes as actions that “slightly or significantly alter the world’s development trajectory.”
What Does the EV Need to be to Prioritize Extinction Risks?
Explosive forces, energy, materials, machinery will be available upon a scale which can annihilate whole nations. Despotisms and tyrannies will be able to prescribe the lives and even the wishes of their subjects in a manner never known since time began. If to these tremendous and awful powers is added the pitiless sub-human wickedness which we now see embodied in one of the most powerful reigning governments, who shall say that the world itself will not be wrecked, or indeed that it ought not to be wrecked? There are nightmares of the future from which a fortunate collision with some wandering star, reducing the earth to incandescent gas, might be a merciful deliverance. ⸻ Winston Churchill (1931)
Under the standard definition of utility, you should take actions with positive expected value (EV), not take actions with negative EV, and it doesn’t matter if you take actions with zero EV. However, prioritization is plausibly much more complicated than this. Is the EV of the action higher than counterfactual actions? Is EV the right approach for imperfect individual decision-makers? Is EV the right approach for a group of people working together? What is the track record for EV decision-making relative to other approaches? Etc. There are many different views that a reasonable person can come to on how best to navigate these conceptual and empirical questions, but I believe that the EV needs to be highly positive to prioritize extinction risks.
As I discussed in Anthis (2018b), I think an intuitive but mistaken argument on this topic is that if we are uncertain about the EV or expect it is close to zero, we should favor reducing extinction risk to preserve option value. Fortunately I have heard this argument much less frequently in recent years, but it is still in a drop-down section of 80,000 Hours’ “The Case for Reducing Existential Risks.” This reasoning seems mistaken for two reasons:
First, option value is only good insofar as we have control over the exercising of future options or expect those who have control to exercise it well. In the course of human civilization, even the totality of the EA movement has relatively little control over humanity’s actions—though arguably a lot more than most measures would make it appear due to our strategic approach, particularly targeting high-leverage domains such as advanced AI—and it is unclear that EA will retain even this modest level of control. The argument that option value is good because our descendants will use it well is circular because the case against extinction risk reduction is primarily focused on humanity not using its options well (i.e., humanity not using its options well is both the premise and the conclusion). An argument that relies on the claim that is being contested is very limited. However, we have more control if one thinks extinction timelines are very short and, if one survives, they and their colleagues will have substantial control over humanity’s actions; we also may be optimistic about human action despite being pessimistic about the future if we think nonhuman forces such as aliens and evolution are the decisive drivers of long-term disvalue.
Second, continued human existence very plausibly limits option value in similar ways to nonexistence. Whether we are in a time of perils or not, there is no easy “off switch” for which humanity can decide to let itself go extinct, especially with advanced technologies (e.g., spreading out through von Neumann probes). It is not as if we can or should reduce extinction risk in the 2020s then easily raise it in the 2030s based on further global priorities research. Still, there is a greater variety of non-extinct than extinct civilizations, so insofar as we want to preserve a wide future of possibilities, that is reason to favor extinction risk reduction.
Instead of option value, the more important considerations to me are (i) that we have other promising options with high EV such that extinction risk reduction needs to be more positive than these other options in order to justify prioritization and (ii) that we should have some risk aversion and sandboxing of EV estimates such that we should sometimes treat close-to-zero values as zero. It’s also unclear how to weigh the totality of evidence here, but insofar as it is weak and speculative—as with most questions about the long-term future—one may pull their estimate towards a prior, though it is unclear what that prior should be. If one thinks zero is a particularly common answer in an appropriate reference class, that could be reasonable, but it depends on many factors beyond the scope of this essay.
Time-Sensitivity
If we are allocating resources to both population and quality risks, one could argue that we should spend resources on population risks first because the quality of individual lives only matters insofar as those individuals exist. The opposite is true as well: For example, if a quality of zero were locked in for the long-term future, then increasing or decreasing the population would have no moral value or disvalue. Outcomes of exactly zero quality might seem less likely than outcomes of exactly zero population, though this depends on the “EV of the Counterfactual” (e.g., life originating on other planets) and is more contentious for close-to-zero quantities.
As with option value, the future depends on the past, so for every year that passes, the future has fewer degrees of freedom. This is most apparent in the development of advanced AI, in which its development may hinge on early-stage choices, such as selecting training regimes that are more likely to lead to its alignment with its designers’ value or selecting those values with which to align the AI (i.e., value lock-in). In general, there are strong arguments for time-sensitivity for both types of trajectory change, especially with advanced technology—also life extension and von Neumann probes in particular.
Biases
To our amazement we suddenly exist, after having for countless millennia not existed; in a short while we will again not exist, also for countless millennia. That cannot be right, says the heart. ⸻ Arthur Schopenhauer (1818, translation 2008)
We could be biased towards optimism or pessimism. Among the demographics of EA, I think that we should probably be more worried about bias towards optimism. Extreme suffering, as described by Tomasik (2006), is a topic that people are very tempted to ignore, downplay, or rationalize (Cohen 2001). In general, the prospect of future dystopias is uncomfortable and unpleasant to think about. Most of us dread the possibility that our legacy in the universe could be a tragic one, and such a gloomy outlook does not resonate with favored trends of techno-optimism or the heroic notion of saving humanity from extinction. However, the sign of this bias can be flipped, such as in social groups where pessimism and doomsaying is in vogue. My experience is that people in EA and longtermism tend to be much more ready to dismiss pessimism and suffering-focused ethics than optimism and happiness-focused ethics, especially based on superficial claims that pessimism is driven by the personal dispositions and biases of its proponents. For a more detailed discussion on biases related to (not) prioritizing suffering, see Vinding (2020).
Additionally, given the default approach to longtermism and existential risk is to reduce extinction risk, and there has already been over a decade of focus on that, we should be very concerned about status quo bias and the incentive structure of EA as it is today. This is one reason to encourage self-critique as individuals and as a community, such as with the Criticism and Red-Teaming Contest. That contest is one reason I wrote this essay, though I had already committed to writing a book chapter on this topic before the contest was announced.
I think we should focus more on the object-level arguments than on biases, but given how our answer to this question hinges on our intuitive estimates of extremely complicated figures, bias is probably more important than normal. I further discussed the merits of considering bias and listed many possible biases towards both moral circle expansion and reducing extinction risk through AI alignment in Anthis (2018b).
One conceptual challenge is that a tendency towards pessimism or optimism could either be accounted for as a bias that needs correction or as a fact about the relative magnitudes of value and disvalue. On one hand, we might say that the importance of disvalue in evolution (e.g. the constant danger of one misstep curtailing all future spread of one’s genes) has made us care more about suffering than we should. On the other hand, we might say that it is a fact about how disvalue tends to be more common, subjectively worse, or objectively worse in the universe.
Future Research on the EV of Human Expansion
Because most events in the long-term future entail some sort of value or disvalue, most new information from longtermist research provides some evidence on the EV of human expansion. As stated above, I’m particularly excited about cooperative game theory research (e.g., CLR, CHAI), moral circle expansion and digital minds research (e.g., SI, FHI), and exploration of concrete trajectories (e.g., Hanson 2016; Hubinger 2021). I’m relatively less excited (though still excited!), on the margin, by entirely armchair taxonomization and argumentation like that in this essay. That includes research on axiological asymmetries, such as more debate on suffering-focused ethics or population ethics, though these can be more useful for other topics and perhaps other people considering this question. My lack of enthusiasm is largely because in the past 8 years of having this view that the EV of human expansion is not highly positive, very little of the new evidence has come from armchair reasoning and argumentation, despite that being more common (although what sort of research is most common depends on where one draws the boundaries because, again, so much research has implications for EV).
In general, this is such an encompassing, big-picture topic that empirical data is extremely limited relative to scope, and it seems necessary to rely on qualitative intuitions, quantitative intuitions, or back-of-the-envelope calculations a la Dickens’ (2016) “A Complete Quantitative Model for Cause Selection” or Tarsney’s (2022) “The Epistemic Challenge to Longtermism.” I would like to see a more systematic survey of such intuitions, ideally from 5-30 people who have read through this essay and the “Related Work.” Ideally these would be stated as credible intervals or similar probability distributions, such that we can more easily quantify uncertainty in the overall estimate. As with all topics, I think we should Aumann update on each other’s views, a process in which I split the difference between my belief and someone else’s even if I do not know all the prior and posterior evidence on which they base their view. Of course, this is messy in the real world, for instance because we presumably should account not just for the few people with whom we happen to know their beliefs, but also for our expectations of the many people who also have a belief and even hypothetical people who could have a belief (e.g., unbiased versions of real-world people). It is also unclear whether normative (e.g., moral) views constitute the sort of belief that should be updated in this way, such as between people with fundamentally different value trade-offs between happiness and suffering.[7] There are cooperative reasons to deeply account for others’ views, and one may choose to account for moral uncertainty.[6] In general, I would be very interested in a survey that just asks for numbers like those in the table above and allows us to aggregate those beliefs in a variety of ways; a more detailed case for how that aggregation should work is beyond the scope of this essay.
If you are persuaded by the arguments that the expected value of human expansion is not highly positive or that we should prioritize the quality of the long-term future, promising approaches include research, field-building, and community-building, such as at the Center on Long-Term Risk, Center for Reducing Suffering, Future of Humanity Institute, Global Catastrophic Risk Institute, Legal Priorities Project, and Open Philanthropy, and Sentience Institute, as well as working at other AI safety and EA organizations with an eye towards ensuring that, if we survive, the universe is better for it. Some of this work has substantial room for more funding, and related jobs can be found at these organizations’ websites and on the 80,000 Hours job board.
References
Aird, Michael. 2020a. “Clarifying Existential Risks and Existential Catastrophes.” Effective Altruism Forum. https://forum.effectivealtruism.org/posts/skPFH8LxGdKQsTkJy/clarifying-existential-risks-and-existential-catastrophes.
———. 2020b. “Venn Diagrams of Existential, Global, and Suffering Catastrophes.” Effective Altruism Forum. https://forum.effectivealtruism.org/posts/AJbZ2hHR4bmeZKznG/venn-diagrams-of-existential-global-and-suffering.
Althaus, David, and Tobias Baumann. 2020. “Reducing Long-Term Risks from Malevolent Actors.” Effective Altruism Forum. https://forum.effectivealtruism.org/posts/LpkXtFXdsRd4rG8Kb/reducing-long-term-risks-from-malevolent-actors.
Althaus, David, and Lukas Gloor. 2016. “Reducing Risks of Astronomical Suffering: A Neglected Priority.” Center on Long-Term Risk. https://longtermrisk.org/reducing-risks-of-astronomical-suffering-a-neglected-priority/.
Anthis, Jacy Reese. 2014. “How Do We Reliably Impact the Far Future?” The Best We Can. https://web.archive.org/web/20151106103159/http://thebestwecan.org/2014/07/20/how-do-we-reliably-impact-the-far-future/.
———. 2016a. “Some Considerations for Different Ways to Reduce X-Risk.” Effective Altruism Forum. https://forum.effectivealtruism.org/posts/NExT987oY5GbYkTiE/some-considerations-for-different-ways-to-reduce-x-risk.
———. 2016b. “Why Animals Matter for Effective Altruism.” Effective Altruism Forum. https://forum.effectivealtruism.org/posts/ch5fq73AFn2Q72AMQ/why-animals-matter-for-effective-altruism.
———. 2018a. The End of Animal Farming: How Scientists, Entrepreneurs, and Activists Are Building an Animal-Free Food System. Boston: Beacon Press.
———. 2018b. “Why I Prioritize Moral Circle Expansion Over Artificial Intelligence Alignment.” Effective Altruism Forum. https://forum.effectivealtruism.org/posts/BY8gXSpGijypbGitT/why-i-prioritize-moral-circle-expansion-over-artificial.
———. 2018c. “Animals and the Far Future.” EAGxAustralia. https://www.youtube.com/watch?v=NTV81NZSuKw.
———. 2022. “Consciousness Semanticism: A Precise Eliminativist Theory of Consciousness.” In Biologically Inspired Cognitive Architectures 2021, edited by Valentin V. Klimov and David J. Kelley, 1032:20–41. Studies in Computational Intelligence. Cham: Springer International Publishing. https://doi.org/10.1007/978-3-030-96993-6_3.
Anthis, Jacy Reese, and Eze Paez. 2021. “Moral Circle Expansion: A Promising Strategy to Impact the Far Future.” Futures 130: 102756. https://doi.org/10.1016/j.futures.2021.102756.
Askell, Amanda, Yuntao Bai, Anna Chen, et al. “A General Language Assistant as a Laboratory for Alignment.” ArXiv. https://arxiv.org/abs/2112.00861.
Beckstead, Nick. 2013a. “On the Overwhelming Importance of Shaping the Far Future.” Rutgers University. https://doi.org/10.7282/T35M649T.
———. 2013b. “A Proposed Adjustment to the Astronomical Waste Argument.” effectivealtruism.org. https://www.effectivealtruism.org/articles/a-proposed-adjustment-to-the-astronomical-waste-argument-nick-beckstead.
Benatar, David. 2006. Better Never to Have Been: The Harm of Coming into Existence. New York: Clarendon Press.
Bostrom, Nick. 2002. “Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards.” Journal of Evolution and Technology 9. https://ora.ox.ac.uk/objects/uuid:827452c3-fcba-41b8-86b0-407293e6617c.
———. 2003. “Astronomical Waste: The Opportunity Cost of Delayed Technological Development.” Utilitas 15 (3): 308–14. https://doi.org/10.1017/S0953820800004076.
———. 2003. “Moral uncertainty – towards a solution?” Overcoming Bias. https://www.overcomingbias.com/2009/01/moral-uncertainty-towards-a-solution.html.
———. 2012. Global Catastrophic Risks. Repr. Oxford: Oxford University Press.
———. 2013. “Existential Risk Prevention as Global Priority.” Global Policy 4 (1): 15–31. https://doi.org/10.1111/1758-5899.12002.
———. 2014. Superintelligence: Paths, Dangers, Strategies. Oxford: Oxford University Press.
Bradbury, Ray. 1979. “Beyond 1984: The People Machines.” In Yestermorrow: Obvious Answers to Impossible Futures.
Brauner, Jan M., and Friederike M. Grosse-Holz. 2018. “The Expected Value of Extinction Risk Reduction Is Positive.” Effective Altruism Forum. https://forum.effectivealtruism.org/posts/NfkEqssr7qDazTquW/the-expected-value-of-extinction-risk-reduction-is-positive?fbclid=IwAR2Si8qdOEqXdPujDfv6gDGLaTdevs4Tb_CALW0D2MHUC4Ot9evEAoem3Gw.
Christiano, Paul. 2013. “Why Might the Future Be Good?” Rational Altruist. https://rationalaltruist.com/2013/02/27/why-will-they-be-happy/.
Churchill, Winston. 1931. “Fifty Years Hence. https://www.nationalchurchillmuseum.org/fifty-years-hence.html.
Cicero, Marcus Tullius. 1931. Cicero: De Finibus. Translated by H. Harris Rackham. 2nd ed. Vol. XVII. Loeb Classical Library. Cambridge: Harvard University Press.
Cowen, Tyler. 2018. Stubborn Attachments: A Vision for a Society of Free, Prosperous, and Responsible Individuals.
Crootof, Rebecca. 2019. “'Cyborg Justice' and the Risk of Technological-Legal Lock-In.” _119 Columbia Law Review Forum _233.
Deutsch, David. 2011. The Beginning of Infinity: Explanations That Transform the World. London: Allen Lane.
Dickens, Michael. 2016. “A Complete Quantitative Model for Cause Selection.” Effective Altruism Forum. https://forum.effectivealtruism.org/posts/fogJKYXvqzkr9KCud/a-complete-quantitative-model-for-cause-selection.
DiGiovanni, Anthony. 2021. “A Longtermist Critique of ‘The Expected Value of Extinction Risk Reduction Is Positive.’” Effective Altruism Forum. https://forum.effectivealtruism.org/posts/RkPK8rWigSAybgGPe/a-longtermist-critique-of-the-expected-value-of-extinction-2.
Gloor, Lukas. 2017. “Tranquilism.” Center on Long-Term Risk. https://longtermrisk.org/tranquilism/.
———. 2018. “Cause Prioritization for Downside-Focused Value Systems.” Effective Altruism Forum. https://forum.effectivealtruism.org/posts/225Aq4P4jFPoWBrb5/cause-prioritization-for-downside-focused-value-systems.
Greaves, Hilary, and Will MacAskill. 2017. “A Research Agenda for the Global Priorities Institute.” https://globalprioritiesinstitute.org/wp-content/uploads/GPI-Research-Agenda-December-2017.pdf.
Hanson, Robin. 2016. The Age of Em: Work, Love, and Life When Robots Rule the Earth. First Edition. Oxford: Oxford University Press.
Harris, Jamie. 2019. “How Tractable Is Changing the Course of History?” Sentience Institute. http://www.sentienceinstitute.org/blog/how-tractable-is-changing-the-course-of-history.
Hobbhan, Marius, Eric Landgrebe, and Beth Barnes. “Reflection Mechanisms as an Alignment target: A Survey.” LessWrong. https://www.lesswrong.com/posts/XyBWkoaqfnuEyNWXi/reflection-mechanisms-as-an-alignment-target-a-survey-1.
Hubinger, Evan. 2021. “How Do We Become Confident in the Safety of a Machine Learning System? - AI Alignment Forum.” AI Alignment Forum. https://www.alignmentforum.org/posts/FDJnZt8Ks2djouQTZ/how-do-we-become-confident-in-the-safety-of-a-machine.
Knutsson, Simon. 2017. “Reply to Shulman’s ‘Are Pain and Pleasure Equally Energy-Efficient?’” http://www.simonknutsson.com/reply-to-shulmans-are-pain-and-pleasure-equally-energy-efficient/.
MacAskill, William. Forthcoming (2022). What We Owe the Future: A Million-Year View. New York: Basic Books.
Matheny, Jason G. 2007. “Reducing the Risk of Human Extinction.” Risk Analysis 27 (5): 1335–44. https://doi.org/10.1111/j.1539-6924.2007.00960.x.
Moynihan, Thomas. 2020. X-Risk: How Humanity Discovered Its Own Extinction. Falmouth: Urbanomic.
Ord, Toby. 2020. The Precipice: Existential Risk and the Future of Humanity. New York: Hachette Books.
Parfit, Derek. 2017. On What Matters: Volume Three. Oxford: Oxford University Press.
Pinker, Steven. 2012. The Better Angels of Our Nature. New York Toronto London: Penguin Books.
———. 2018. Enlightenment Now. New York, New York: Viking, an imprint of Penguin Random House LLC.
Plant, Michael. 2022. “Will faster economic growth make us happier? The relevance of the Easterlin Paradox to Progress Studies.” Effective Altruism Forum. https://forum.effectivealtruism.org/posts/gCDsAj3K5gcZvGgbg/will-faster-economic-growth-make-us-happier-the-relevance-of.
Rowe, Abraham. 2022. “Critiques of EA that I Want to Read.” Effective Altruism Forum. https://forum.effectivealtruism.org/posts/n3WwTz4dbktYwNQ2j/critiques-of-ea-that-i-want-to-read.
Russell, Stuart J. 2019. Human Compatible: Artificial Intelligence and the Problem of Control. New York: Viking.
Schopenhauer, Arthur. 2008 [1818]. The World as Will and Representation. New York: Routledge.
Shulman, Carl. 2012. “Are Pain and Pleasure Equally Energy-Efficient?” Reflective Disequillibrium. http://reflectivedisequilibrium.blogspot.com/2012/03/are-pain-and-pleasure-equally-energy.html.
Smith, Tom W., Peter Marsden, Michael Hout, and Jibum Kim. 2022. “General Social Surveys, 1972-2022.” National Opinion Research Center. https://www.norc.org/PDFs/COVID Response Tracking Study/Historic Shift in Americans Happiness Amid Pandemic.pdf.
Tarsney, Christian. 2022. “The Epistemic Challenge to Longtermism.” Global Priorities Institute. https://globalprioritiesinstitute.org/wp-content/uploads/Tarsney-Epistemic-Challenge-to-Longtermism.pdf.
Tegmark, Max. 2017. Life 3.0: Being Human in the Age of Artificial Intelligence. New York: Alfred A. Knopf.
Tomasik, Brian. 2006. “On the Seriousness of Suffering.” Essays on Reducing Suffering. https://reducing-suffering.org/on-the-seriousness-of-suffering/.
———. 2011. “Risks of Astronomical Future Suffering.” Foundational Research Institute. https://foundational-research.org/risks-of-astronomical-future-suffering/.
———. 2013a. “The Future of Darwinism.” Essays on Reducing Suffering. https://reducing-suffering.org/the-future-of-darwinism/.
———. 2013b. “Values Spreading Is Often More Important than Extinction Risk.” Essays on Reducing Suffering. https://reducing-suffering.org/values-spreading-often-important-extinction-risk/.
———. 2014. “Why the Modesty Argument for Moral Realism Fails.” Essays on Reducing Suffering. https://reducing-suffering.org/why-the-modesty-argument-for-moral-realism-fails/.
———. 2015. “Artificial Intelligence and Its Implications for Future Suffering.” Center on Long-Term Risk. https://longtermrisk.org/artificial-intelligence-and-its-implications-for-future-suffering/.
———. 2017. “Will Future Civilization Eventually Achieve Goal Preservation?” Essays on Reducing Suffering. https://reducing-suffering.org/will-future-civilization-eventually-achieve-goal-preservation/.
Vinding, Magnus. 2020. Suffering-Focused Ethics: Defense and Implications. Ratio Ethica.
West, Ben. 2017. “An Argument for Why the Future May Be Good.” Effective Altruism Forum. https://forum.effectivealtruism.org/posts/kNKpyf4WWdKehgvRt/an-argument-for-why-the-future-may-be-good.
Wolf, Clark. 1997. “Person-Affecting Utilitarianism and Population Policy; or, Sissy Jupe’s Theory of Social Choice.” In Contingent Future Persons, eds. Nick Fotion and Jan C. Heller. Dordrecht: Springer Dordrecht. https://doi.org/10.1007/978-94-011-5566-3_9.
Yudkowsky, Eliezer. 2004. “Coherent Extrapolated Volition.” The Singularity Institute. https://intelligence.org/files/CEV.pdf.
———. 2007. “The Hidden Complexity of Wishes.” LessWrong. https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden-complexity-of-wishes.
- ^
For the sake of brevity, while I have my own views of moral value and disvalue, I don’t tie this essay to any particular view (e.g., utilitarianism). For example, it can include subjective goods (valuable for a person) and objective goods (valuable regardless of people), and it can be understood as estimates or direct observation of realist good (stance-independent) or anti-realist good (stance-dependent). Some may also have moral aims aside from maximizing expected “value” per se, at least for certain senses of “expected” and “value.” There is a substantial philosophical literature on such topics that I will not wade into, and I believe such non-value-based arguments can be mapped onto value-based arguments with minimal loss (e.g., not having a duty to make happy people can be mapped onto there being no value in making happy people).
- ^
Both population risks and quality risks can be existential risks—though longtermist EAs have usually defaulted to a focus on population risks, particularly extinction risks.
- ^
For the sake of brevity, I analyze human survival and interstellar colonization together under the label “human expansion.” I gloss over possible futures in which humanity survives but does not colonize space.
- ^
For example, the portion of historical progress made through market mechanisms is split among Historical Progress insofar as this is a large historical trend, Value Through Intent insofar as humans intentionally progressed in this way, Value Through Evolution insofar as selection increased the prevalence of these mechanisms, and Reasoned Cooperation insofar as the intentional change was through reasoned cooperation. How is this splitting calculated? I punt to future work, but in general, I mean some sort of causal attribution measure. For example, if I grow an apple tree that is caused by both rain and soil nutrients, then I would assign more causal force to rain if and only if reducing rain by one standard deviation would inhibit growth more than reducing soil nutrients by one standard deviation. Related measures include Shapley values and LIME.
- ^
I do not provide specific explanations for the weights in the spreadsheet because they are meant as intuitive, subjective estimates of the linear weight of the argument as laid out in the description column. As discussed in the “Future Research on the EV of Human Expansion” subsection, unpacking these weights into probability distributions and back-of-the-envelope estimates is a promising direction for better estimating the EV of human expansion. The evaluations rely on a wide range of empirical, conceptual, and intuitive evidence. These numbers should be taken with many grains of salt, but as the “superforecasting” literature evidences, it can be useful to quantify seemingly hard-to-quantify questions. The weights in this table are meant as linear, and the linear sum is -7. There are many approaches we could take to aggregating such evidence, reasoning, and intuitions; we could entirely avoid quantification entirely and take the gestalt of these arguments. If taken as logarithms of 2 (e.g., take 0 as 0, take 1 as 2, take 10 as 2^10=1024) as the prior that EA arguments tend to vary in weight by doubling rather than linear scaling, then the mean is -410. Again, these are just two of the many possible ways to aggregate arguments on this topic. Also, for methodological clarity at the risk of droning, I assign weights constantly across arguments (e.g., 2 arguments of weight +2 are the same evidential weight as 4 arguments of weight +1), though other assignment methods are reasonable, and again, other divisions of the arguments (i.e., other numbers of rows in the table) are reasonable and would make no difference in my own additive total, though they could change the logarithmic total and some other aggregations.
- ^
While this is a very contentious view among some in EA, I should note that I’m not persuaded by, and I don’t account for, moral uncertainty because I don’t think a “Discoverable Moral Reality” is plausible, and I doubt I would be persuaded to act in accordance with it if it did exist (e.g., to cause suffering if suffering were stance-independently good)—though it is unclear what it would even mean for a vague, stance-independent phenomenon to exist (Anthis 2022). Moreover, I’m not compelled by arguments to account for any sort of anti-realist moral uncertainty, views which are arguably better not even described as “uncertainty” (e.g., weighting my future self’s morals, such as after a personality-altering brain injury or taking rationality- and intelligence-increasing nootropics; across different moral frameworks, such as in a Bostrom's (2009) “moral parliament”). Of course, I still account for moral cooperation and standard empirical uncertainty.
- ^
There is much more to say about how Aumann’s Agreement Theorem obtains in the real world than what I have room for here. For example, Andrew Critch states that the “common priors” assumption “seems extremely unrealistic for the real world.” I’m not sure if I disagree with this, but when I describe Aumann updating, I’m not referring to a specific prior-to-posterior Bayesian update; I’m referring to the equal treatment of all the evidence going into my belief with all the evidence going into my interlocutor’s belief. If nothing else, this can be viewed as an aggregation of evidence in which each agent is still left with aggregating their evidence and prior, but I don’t like approaching such questions with a bright line between prior and posterior except in a specific prior-to-posterior Bayesian update (e.g., You believe the sky is blue but then walk outside one day and see it looks red; how should this change your belief?).
I'm shocked and somewhat concerned that your empirical finding is that so few people have encountered or thought about this crucial consideration.
My experience is different, with maybe 70% of AI x-risk researchers I've discussed with being somewhat au fait with the notion that we might not know the sign of future value conditional on survival. But I agree that it seems people (myself included) have a tendency to slide off this consideration or hope to defer its resolution to future generations, and my sample size is quite small (a few dozen maybe) and quite correlated.
For what it's worth, I recall this question being explicitly posed in at least a few of the EA in-depth fellowship curricula I've consumed or commented on, though I don't recall specifics and when I checked EA Cambridge's most recent curriculum I couldn't find it.
My anecdata is also that most people have thought about it somewhat, and "maybe it's okay if everyone dies" is one of the more common initial responses I've heard to existential risk.
But I agree with OP that I more regularly hear "people are worried about negative outcomes just because they themselves are depressed" than "people assume positive outcomes just because they themselves are manic" (or some other cognitive bias).
My impression was that due to multiple accusations of sexual harassment, Jacy Reese Anthis was stepping back from the community. When and why did this stop?
He was evicted from Brown University in 2012 for sexual harassment (as discussed here).
And he admitted to several instances of sexual harassment (as discussed here).
He also lied on his website about being a founder of effective altruism.
Some notes from CEA:
Is there more information you can share without risking the anonymity of the complainants or victims? E.g.,
I can understand why people want more info. Jacy and I agreed three years ago what each of us would say publicly about this, and I think it would be difficult and not particularly helpful to revisit the specifics now.
If anyone is making a decision where more info would be helpful, for example you’re deciding whether to have him at an event or you’re running a community space and want to think about good policies in general, please feel free to contact me and I’ll do what I can to help you make a good decision.
For convenience, this is CEA's statement from three years ago:
By my reading, the information about the reports contained in this is:
- CEA received reports from several parties about concerns over Jacy's behavior over several time periods
- CEA found the reports 'credible and concerning'
- CEA cannot discuss details of most of these concerns because the people who raised them want to protect their confidentiality
- It also implies that Jacy did not treat people with fairness and respect in the reported incidents
- 'It’s very important to CEA tha
... (read more)I have a general objection to this, but I want to avoid getting entirely off topic. So I'll just say, this seems to me to only shift the problem further away from the people affected.
I've also been surprised to see Jacy engaging publicly with the EA community again recently, without any public communication about what's changed.
I downvoted this comment. While I think this discussion is important to have, I do not think that a post about longtermism should be turned into a referendum on Jacy's conduct. I think it would be better to have this discussion on a separate post or the open thread.
We don't have any centralized or formal way of kicking people out of EA. Instead, the closest we have, in cases where someone has done things that are especially egregious, is making sure that everyone who interacts with them is aware. Summarizing the situation in the comments here, on Jacy's first EA forum post in 3 years (Apology, 2019-03), accomplishes that much more than posting in the open thread.
This is a threaded discussion, so other aspects of the post are still open to anyone interested. Personally, I don't think Jacy should be in the EA movement and won't be engaging in any of the threads below.
But what about the impact on the topic itself? Having the discussion heavily directed to a largely irrelevant topic, and affecting its down/upvoting situation, doesn't do the original topic justice. And this topic could potentially be very important for the long-term future.
I think that's a strong reason for people other than Jacy to work on this topic.
Watching the dynamic here I suspect this might likely be true. But I would still like to point out that there should be a norm about how these situations should be handled. This likely won't be the last EA forum post that goes this way.
To be honest I am deeply disappointed and very worried that this post has gone this way. I admit that I might be feeling so because I am very sympathetic to the key views described in this post. But I think one might be able to imagine how they feel if certain monumental posts that are crucial to the causes/worldviews they care dearly about, went this way.
Relevant with respect to what? For me, the most sensible standard to use here seems to be "whether it is relevant to the original topic of the post (the thesis being brought up, or its antithesis)". Yes, the topic of personal behavior is relevant to EA's stability and therefore how much good we can do, or even the long-term future. But considering that there are other ways of letting people know what is being communicated here, such as starting a new post, I don't think we should use this criterion of relevance.
That's true, logically speaking. But that's also logically true for EA/EA-like communities. In other words, it's always "possible" that if this EA breaks, there "could be" another similar one that will be formed again. But I am guessing not many people would like to take the bet based on the "come again argument". Then what is our reason for being willing to take a similar bet with this potentially important - I believe crucial - topic (or just any topic)?
And again, the fact that there are other ways to bring up the to... (read more)
But wouldn't a new post on this topic serve the same purpose of expressing and discussing this concern, without having the effects of affecting this topic?
A comment from the moderation team:
This topic is extremely difficult to discuss publicly in a productive way. First, a lot of information isn’t available to everyone — and can’t be made available — so there’s a lot of guesswork involved. Second, there are a number of reasons to be very careful; we want community spaces to be safe for everyone, and we want to make sure that issues with safety can be brought up, but we also require a high level of civility on this Forum.
We ask you to keep this in mind if you decide to contribute to this thread. If you’re not sure that you will contribute something useful, you might want to refrain from engaging. Also, please note that you can get in touch with the Community Health team at CEA if you’d like to bring up a specific concern in a less public way.
I recommend a mediator be hired to work with Jacy and whichever stakeholders are relevant (speaking broadly). This will be more productive than a he-said she-said forum discussion that is very emotionally toxic for many bystanders.
I'm confused that you seem to claim strong evidence on the basis on a variety of things that seem like weak evidence to me. While I am sure details should not be provided, can you clarify whether you have non-public information about what happened post 2016 that contradicts what Kelly and Jacy have said publicly about it?
Thanks for writing this.
As everyone here knows, there has been an influx of people into EA and the forum in the last couple years, and it seems probable that most of the people here (including me) wouldn't have known about this if not for this reminder.
Jacy Reese claims that the allegations discussed in the Forum post centre on 'clumsy online flirting'. We don't really know what the allegations are, but CEA :
We have zero reason to believe Jacy about the substance of the allegations, given his documented history of lying and incentives to lie in the case.
I don’t think (or, you have not convinced me that) it’s appropriate to use CEA’s actions as strong evidence against Jacy. There are many obvious pragmatic justifications to do so that are only slightly related to the factual basis of the allegations—I.e., even if the allegations are unsubstantiated, the safest option for a large organization like CEA would be to cut ties with him regardless. Furthermore, saying someone has “incentives to lie” about their own defense also feels inappropriate (with some exceptions/caveats), since that basically applies to almost every situation where someone has been accused. The main thing that you mentioned which seems relevant is his “documented history of lying,” which (I say this in a neutral rather than accusatory way) I haven’t yet seen documentation of.
Ultimately, these accusations are concerning, but I’m also quite concerned of the idea of throwing around seemingly dubious arguments in service of vilifying someone.
Please provide specific quotes, I spent a few minutes reading the first part of that without seeing what you were referring to
If you’re referring to the same point about his claim to be a cofounder, I did just see that. However, unless I see some additional and/or more-egregious quotes from Jacy, I have a fairly negative evaluation of your accusation. Perhaps his claim was a bit exaggerative combined with being easily misinterpreted, but it seems he has walked it back. Ultimately, this really does not qualify in my mind as “a history of lying.”
In most cases where I am actually familiar with the facts CEA has behaved very poorly. They have both been way too harsh on good actors and failed to take sufficient action against bad actors (ex Kathy Forth). They did handle some very obvious cases reasonably though (Diego). I don't claim I would do a way better job but I don't trust CEA to make these judgments.
Could you
I still don't see where 'he admitted to several instances of sexual harassment' as you've claimed.
How else would you define the apology post other than an apology for sexual harassment? I would have thought the debate would be about an appropriate time for him to rejoin the community not about whether he actually committed sexual harassment. Or whether he was unfortunate enough for multiple women to independently accuse of him sexual harassment throughout his life
Hi John, just to clarify some inaccuracies in your two comments:
- I’ve never harassed anyone, and I’ve never stated or implied that I have. I have apologized for making some people uncomfortable with “coming on too strong” in my online romantic advances. As I've said before in that Apology, I never intended to cause any discomfort, and I’m sorry that I did so. There have, to my knowledge, been no concerns about my behavior since I was made aware of these concerns in mid-2018.
- I didn’t lie on my website. I had (in a few places) described myself as a “co-founder” of EA [Edit: Just for clarity, I think this was only on my website for a few weeks? I think I mentioned it and was called it a few times over the years too, such as when being introduced for a lecture. I co-founded the first dedicated student group network, helped set up and moderate the first social media discussion groups, and was one of the first volunteers at ACE as a college student. I always favored a broader-base view of how EA emerged than what many perceived at the time (e.g., more like the founders of a social movement than of a company). Nobody had pushed back against "co-founder" until 2019, an... (read more)
Hi Jacy, you said in your apology "I am also stepping back from the EA community more generally, as I have been planning to since last year in order to focus on my research."
I haven't seen you around since then, so was surprised to see you attend an EA university retreat* and start posting more about EA. Would you describe yourself as stepping back into the EA community now?
*https://twitter.com/jacyanthis/status/1515682513280282631?s=20&t=reRvYxXCs2z-AvszF31Gng
Hi Khorton, I wouldn't describe it as stepping back into the community, and I don't plan on doing that, regardless of this issue, unless you consider occasional posts and presentations or socializing with my EA friends as such. This post on the EV of the future was just particularly suited for the EA Forum (e.g., previous posts on it), and it's been 3 years since I published that public apology and have done everything asked of me by the concerned parties (around 4 years since I was made aware of the concerns, and I know of no concerns about my behavior since then).
I'm not planning to comment more here. This is in my opinion a terrible place to have these conversations, as Dony pointed out as well.
I wouldn't have described Jacy as a co-founder of effective altruism and don't like him having had it on his website, but it definitely doesn't seem like a lie to me (I kind of dislike the term "co-founder of EA" because of how ambiguous it is).
Anyway I think calling it a lie is roughly as egregious a stretch of the truth as Jacy's claim to be a co-founder (if less objectionable since it reads less like motivated delusion). In both cases I'm like "seems wrong to me, but if you squint you can see where it's coming from".
[meta for onlookers: I'm investing more energy into holding John to high standards here than Jacy because I'm more convinced that John is a good faith actor and I care about his standards being high. I don't know where Jacy is on the spectrum from "kind of bad judgement but nothing terrible" to "outright bad actor", but I get a bad smell from the way he seems to consistently turns to present things in a way that puts him in a relatively positive light and ignores hard questions, so absent further evidence I'm just not very interested in engaging]
It's from "man things in the world are typically complicated, and I haven't spent time digging into this, but although there surface level facts look bad I'm aware that selective quoting of facts can give a misleading impression".
I'm not trying to talk you out of the bad actor categorization, just saying that I haven't personally thought it through / investigated enough that I'm confident in that label. (But people shouldn't update on my epistemic state! It might well be I'd agree with you if I spent an hour on it; I just don't care enough to want to spend that hour.)
Yes, I personally want to do that, because I want to spend time engaging with good faith actors and having them in gated spaces I frequent.
In general I have a strong perfectionist streak, which I channel only to try to improve things which are good enough to seem worth the investment of effort to improve further. This is just one case of that.
(Criticizing is not itself something that comes with direct negative effects. Of course I'd rather place larger sanctions on bad faith actors than good faith actors, but I don't think criticizing should be understood as a form of sanctioning.)
I actually didn't mean for any of my comments here to get into attacks on our defence of Jacy. I don't think I have great evidence and don't think I'm a very good person to listen to on this! I just wanted to come and clarify that my criticism of John was supposed to be just that, and not have people read into it a defence of Jacy.
(I take it that the bar for deciding personally to disengage is lower than for e.g. recommending others do that. I don't make any recommendations for others. Maybe I'll engage with Jacy later; I do feel happier about recent than old evidence, but it hasn't yet moved me to particularly wanting to engage.)
So, are you saying it is an honest mistake but not a lie? His argument for being a co-founder seems to be that he was involved in the utilitarian forum Felicifia in 2008. He didn't even found it. I know several other people who founded or were involved in that forum and none of them has ever claimed to be a founder of effective altruism on that basis. Jacy is the only person to do that and it is clear he does it in order to advance his claim to be a public intellectual because it suggests to the outside world that he was as influential as Will MacAskill, Toby Ord, Elie Hassenfeld, and Holden Karnofsky, which he wasn't and he knows he wasn't.
The dissembling in the post is typical of him. He never takes responsibility for anything unless forced to do so.
I'm saying it's a gross exaggeration not a lie. I can imagine someone disinterested saying "ok but can we present a democratic vision of EA where we talk about the hundred founders?" and then looking for people who put energy early into building up the thing, and Jacy would be on that list.
(I think this is pretty bad, but that outright lying is worse, and I want to protect language to talk about that.)
I want to flag that something like "same intention as outright lying, but doing it in a way to maximize plausible deniability" would be just as bad as outright lying. (It is basically "outright lying" but in a not stupid fashion.)
However, the problem is that sometimes people exaggerate or get things wrong for more innocuous reasons like exaggerated or hyperbolic speech or having an inflated sense of one's importance in what's happening. Those cases are indeed different and deserve to be treated very different from lying (since we'd expect people to self-correct when they get the feedback, and avoid mistakes in the future). So, I agree with the point about protecting language. I don't agree with the implicit message "it's never as bad as outright lying when there's an almost-defensible interpretation somewhere." I think protecting the language is important for reasons of legibility and epistemic transparency, not so much because the moral distinction is always clean-cut.
I agree with this.
You are taking charitable interpretations to an absolute limit here. You seem to be saying "maybe Jacy was endorsing a highly expansive conception of 'founding' which implies that EA has hundreds of founders'". This is indeed a logical possibility. But I think the correct credence to have in this possibility is ~0. Instead, we should have ~1 credence in the following "he said it knowing it is not true in order to further his career". And by 'founding' he meant, "I'm in the same bracket as Will MacAskill". Otherwise, why put it on your website and in your bio?
I don't think it's like "Jacy had an interpretation in mind and then chose statements". I think it's more like "Jacy wanted to say things that made himself look impressive, then with motivated reasoning talked himself into thinking it was reasonable to call himself a founder of EA, because that sounded cool".
(Within this there's a spectrum of more and less blameworthy versions, as well as the possibility of the straight-out lying version. My best guess is towards the blameworthy end of the not-lying versions, but I don't really know.)
This feels off to me. It seems like Jacy deliberately misled people to think that he was a co-founder of EA, to likely further his own career. This feels like a core element of lying, to deceive people for personal gain, which I think is the main reason one would claim they're the co-founder of EA when almost no one else would say this about them.
Sure I think it can also be called "gross exaggeration" but where do you think the line is between "gross exaggeration" and "lying"? For me, lying means you say something that isn't true (in the eyes of most people) for significant personal gain (i.e. status) whereas gross exaggeration is a smaller embellishment and/or isn't done for large personal gain.
It's briefly referenced in this recent post, though I don't think this is what John was talking about.
https://jacyanthis.com/some-early-history-of-effective-altruism
Ah, I only just noticed this had gone online. I think the story Jacy tells is potentially a bit misleading, so I've left some comments here.
You removed it after people called you out on it being bullshit and you know it isn't true.
The moderation team feels that phrases like “called you out on it being bullshit” aren’t constructive for this discussion (or on the Forum as a general rule). Please don’t use them.
I like "quality risks" (q-risks?) and think this is more broadly appealing to people who don't want to think about suffering-reduction as the dominantly guiding frame for whatever reason. Moral trade can be done with people concerned with other qualities, such as worries about global totalitarianism due to reasons independent of suffering such as freedom and diversity.
It's also relatively more neglected than the standard extinction risks, which I am worried we are collectively Goodharting on as our focus (and to a lesser extent, focus on classical suffering risks may fall into this as well). For instance, nuclear war or climate change are blatant and obvious scary problems that memetically propagate well, whereas there may be many q-risks to future value that are more subtle and yet to be evinced.
Tangentially, this gets into a broader crux I am confused by: should we work on obvious things or nonobvious things? I am disposed towards the latter.
Look into suffering-focused AI safety which I think is extremely important and neglected (and s-risks).
Thanks for this! I agree this is a very important question, I sympathize with the view that people overweight some arguments for historical optimism, and I'm mostly on board with the list of considerations. Still, I think your associated EV calculation has significant weaknesses, and correcting for these seems to produce much more optimistic results.
- You put the most weight on historical harms, and you also put a lot of weight on empirical utility asymmetry. But arguably, the future will be deeply different from the past (including through reduced influence of biological evolution), so simple extrapolation from the past or present should not receive very high weight. (For the same reason, we should also downweight historical progress.)
- Arguably, historical harms have occurred largely through the divergence of agency and patiency, so counting both is mostly double-counting. (Similarly, historical progress has largely occurred through the other mechanisms that are already covered.) So we should further downweight these.
- I don't see why we should put negative weight on "The Nature of Digital Minds, People, and Sentience."
- Reasoned cooperation should arguably receive significantly mor
... (read more)I've considered a possible pithy framing of the Life Despite Suffering question as a grim orthogonality thesis (though I'm not sure how useful it is):
We sometimes point to the substantial majority's revealed preference for staying alive as evidence of a 'life worth living'. But perhaps 'staying-aliveness' and 'moral patient value' can vary more independently than that claim assumes. This is the grim orthogonality thesis.
An existence proof for the 'high staying-aliveness x low moral patient value' quadrant is the complex of torturer+torturee, which quite clearly can reveal a preference for staying alive, while quite plausibly being net negative value.
Can we rescue the correlation of revealed 'staying-aliveness' preference with 'life worth livingness'?
We can maybe reason about value from the origin of moral patients we see, without having a physical theory of value. All the patients we see at present are presumably products of natural selection. Let's also assume for now that patienthood comes from consciousness.
Two obvious but countervailing observations
- to the extent that conscious content is upstream of behaviour but downstream of genetic content, natural selection will operate o
... (read more)See also Robert Harling's Good v. Optimal Futures.
I find the following simple argument disturbing:
P1 - Currently, and historically, low power being (animals, children, old dying people) are treated very cruelly if treating them cruelly benefits the powerful even in minor ways. Weak benefits for the powerful empirically justify cruelty at scale.
P2 - There is no good reason to be sure the powerful wont have even minor reasons to be cruel to the powerless (ex: suffering sub-routines, human CEV might include spreading earth like life widely or respect for tradition)
P3 - Inequality between agents is likely to become much more extreme as AI develops
P4 - The scale of potenital suffering will increase by many orders of magnitude
C1 - We are fucked?
Personal Note - There is also no reason to assume me or my loved ones will remain relatively powerful beings
C2 - Im really fucked!
I think considerations like these are important to challenge the recent emphasis on grounding x-risk (really, extinction risk) in near-term rather than long-term concerns. That perspective seems to assume that the EV of human expansion is pretty much settled, so we don’t have to engage too deeply with more fundamental issues in prioritization, and we can instead just focus on marketing.
I’d like to see more written directly comparing the tractability and neglectedness of population risk reduction and quality risk reduction. I wonder if you’ve perhaps overstated things in claiming that a lower EV for human expansion suggests shifting resources to long-term quality risks rather than, say, factory farming. It seems like this claim requires a more detailed comparison between possible interventions.
The thing I have most changed my mind about since writing the post of mine you cite is adjacent to the "disvalue through evolution" category: basically, I've become more worried that disvalue is instrumentally useful. E.g. maybe the most efficient paperclip maximizer is one that's really sad about the lack of paperclips.
There's some old writing on this by Carl Shulman and Brian Tomasik; I would be excited for someone to do a more thorough write up/literature review for the red teaming contest (or just in general).
As a small comment, I believe discussions of consciousness and moral value tend to downplay the possibility that most consciousness may arise outside of what we consider the biological ecosystem.
It feels a bit silly to ask “what does it feel like to be a black hole, or a quasar, or the Big Bang,” but I believe a proper theory of consciousness should have answers to these questions.
We don’t have that proper theory. But I think we can all agree that these megaphenomena involve a great deal of matter/negentropy and plausibly some interesting self-organized microstructure- though that’s purely conjecture. If we’re charting out EV, let’s keep the truly big numbers in mind (even if we don’t know how to count them yet).
Thanks for the post. I've also been surprised how little this is discussed, even though the value of x-risk reduction is almost totally conditional to the answer to this question (the EV of the future conditional on human/progeny survival). Here are my big points to bring up re this issue, though some might be slight rephrasing of yours.
- Interpersonal comparisons of utility canonically have two parts - a definition of utility, of which every sentient being is measured by. Then, to compare and sum utility, one must pick (subjective) weights for each sentient being, scale their utility by the weights, and add everything up. (u_1x_1+.....+u_nx_n). If we don't agree on the weights, it's possible that one person may think the future be in expectation positive while another thinks it will be negative even w/ perfect information of what the future will look like. It could be even harder to agree on the weights of sentient beings when we don't even know what agents are going to be alive. We have obvious candidates for general rules about how to weight utility (brain size, pain receptors, etc.) but who knows how are conceptions of these things will change.
- Basically repeatin
... (read more)Typo hint:
"10<sup>38</sup>" hasn't rendered how you hoped. You can use <dollar>10^{38}<dollar> which renders as 1038
I think that this large argument / counterargument table is a great example of how using a platform like Kialo to better structure debates could be valuable.
Thanks for doing this work but I dont have the patience to read entirely. What is it you found exactly? please put at the top of the summary
It's already in the summary:
Minor technical comment: the links to subsections in the topmost table link to the Google docs version of the article, and I think it would be slightly nicer if they linked to the forum post version.
I think you have undervalued optionality value. Using Ctrl + F I have tried to find and summarise your claims against optionality value:
... (read more)You wrote
"There is a substantial philosophical literature on such topics that I will not wade into, and I believe such non-value-based arguments can be mapped onto value-based arguments with minimal loss (e.g., not having a duty to make happy people can be mapped onto there being no value in making happy people)."
Duty to accomplish X implies much more than an assessment of the value of X. To lack the (moral, legal, or ethical) obligation to bring about a state of affairs does not imply a sense that the state of affairs has no value to you or others.
...