TLDR: We can use probabilistic causal models to compare the expected impact of investing in new ideas to the cost-effectiveness of donating to established charities.

Within the EA community, there is a broad consensus that to do the most good with our limited resources, we have to prioritize what is most cost-effective (Ord, 2019). Identifying the most cost-effective contributions is greatly complicated by the fact that there are many things we don’t know about how much moral value a contribution will create and how much it will cost to achieve that benefit. For some alternatives, there is abundant data from studies that have precisely measured relevant costs and benefits (e.g., randomized controlled trials for certain health interventions), which allow us to use standard existing methods to estimate cost-effectiveness. However, these analyses are heavily biased toward activities with easily measurable, quantitative changes that occur shortly after the intervention in an easily accessible and clearly defined population. As a result, we often do not have estimates of cost-effectiveness for many potentially impactful EA activities--including many activities in other cause areas recommended by 80,000 hours (e.g., improving institutional decision-making)--because several features make the activities less amenable to a randomized control trial, e.g., 

1. They haven’t been done yet.
2. They involve rare or unique contributions.
3. Their primary value lies in generating new knowledge or creating novel interventions
4. Their pathway to impact is somewhat long or indirect. As a consequence, we do not know in advance which specific people will benefit and when they will experience those benefits.
5. The activity targets outcomes other than health or well-being.

This raises the question: how can we best extend cost-effectiveness analysis to activities whose effects on health or well-being are still unknown? One solution (which is the topic of this series of posts) is to develop methods that forecast an intervention’s benefits before they have been measured. We are not the first to notice this issue, and there has already been good work on extending cost-effectiveness analysis to specific activities whose cost-effectiveness has to be predicted, such as biosecurity (Millet & Snyder-Beattie, 2017), existential risk reduction (Bostrom, 2013), preventing the permanent loss of electricity and industrial technology and AI safety (Denkenberger et al., 2021), and research on vaccines (Wildeford, 2018). But, such cost-effectiveness analyses are typically difficult, one-off projects that develop their own methods. They thus lack comparability to other cost-effectiveness analyses and the level of testing and scrutiny as traditional methods co-developed by several independent research groups. We are therefore developing a general, freely available, open-source software infrastructure for conducting predictive cost-effectiveness analyses. We hope this makes predicting the cost-effectiveness of working on the world’s most pressing problems could be made easier, more comparable, and more accurate.

Forecasting the cost-effectiveness of EA activities with probabilistic causal models

Having established that deciding which EA activities are most effective requires forecasting the expected moral value of their consequences, we now outline our general approach to making such forecasts. The essence of our approach is to formulate a probabilistic model of the most important causal chains of events that could lead from the activity to moral value. We then use the causal model to simulate each activity's direct consequences, the consequences of those consequences, and so on, until the causal chain ends in an outcome with known moral value. 

The activities EAs choose between target different outcomes (e.g., community growth vs. poverty reduction). This makes it difficult to compare their effectiveness. To make benefits as different as community growth, existential risk reduction, saving lives, animal welfare, and poverty alleviation comparable, we quantify their moral value in terms of their effect on the sum of all well-being experienced by any sentient being at any time. This includes the suffering and happiness of future generations, as well as the well-being of non-human animals. This way of measuring impact follows from the philosophical stance put forward in What We Owe The Future (MacAskill, 2022) and allows us to quantify the cost-effectiveness of all EA activities in the universal currency of Well-Being Adjusted Life Years per dollar (Frijters et al., 2020; Johnson et al., 2016). 

Our approach, therefore, involves building evidence-based probabilistic causal models of how potential EA activities could impact the sum of all well-being. Our models are evidence-based insofar as we derive the strength and uncertainty about each causal link from the most relevant and reliable data we can find. We then use these models to run thousands of simulations. The outcome of each simulation can be distilled into a single number that signifies the net increase or decrease in well-being caused by the activity in the simulated scenario. The histogram of these numbers provides an estimate of the probability distribution of the activity’s moral value. We can use it to calculate the activity’s expected moral value and the uncertainty entailed by the information we have incorporated into the model. The models we build also include the cost incurred by performing the activity. Dividing the simulated moral value by the cost incurred yields the cost-effectiveness of the intervention in a given scenario. The histogram of these values then provides an estimate of the probability distribution of the activity’s cost-effectiveness. Obtaining these histograms for multiple EA activities then allows us to compare their expected values and compute the probability that a given activity is more cost-effective than any other.

Although valuable, building rigorous, quantitative, evidence-based probabilistic models of the causal pathways through which an EA activity increases future happiness or reduces suffering is also challenging, time-consuming, and error-prone. This might be why this approach is rarely used. Thus, one of our goals is to create software and data infrastructure that will make it easier for cause prioritization researchers and funders to predict and compare the cost-effectiveness of newly proposed EA activities. To create this infrastructure, we build on QURI’s new probabilistic forecasting language Squiggle. We will provide this infrastructure in the form of a modular software library that provides modules that can be easily composed into cost-effectiveness models for a wide range of activities. Those modules will be probabilistic models of causal links or chains that occur in many activities’ pathways to impact. Those modules can then be strung together into the causal chains that describe new EA activities' pathways to impact. We will create evidence-based probabilistic causal models of changes in well-being. These models will quantify the effects of the primary determinants of happiness and suffering. Working backward from there, we will provide causal modules of how the primary determinants of happiness can be changed, and so on. Working forward, we will develop probabilistic causal models of the most relevant immediate and intermediate effects of common types of EA activities. Our library will also include useful data and utility functions. We have started to compile some useful functions at https://observablehq.com/@falk-lieder/cea_library  as we are slowly working towards the eventual release of our library for performing predictive cost-effectiveness analyses.

So far, we have described our approach in very abstract, general terms. To find out if and how it works in practice, stay tuned for the next post in this series, which will apply our method to predict the cost-effectiveness of deploying a new intervention promoting prosocial behavior.
 

References:

1. Bostrom, N. (2013). Existential risk prevention as global priority. Global Policy, 4(1), 15-31. 
2. Denkenberger, D., Sandberg, A., Tieman, R.J., & Pearce, J. M. (2021). Long-term cost-effectiveness of interventions for loss of electricity/industry compared to artificial general intelligence safety. European Journal of Futures Research, 9(11). https://doi.org/10.1186/s40309-021-00178-z
3. Frijters, P., Clark, A., Krekel, C., & Layard, R. (2020). A happy choice: Wellbeing as the goal of government. Behavioural Public Policy, 4(2), 126-165. doi:10.1017/bpp.2019.39 
4. Johnson, R., Jenkinson, D., Stinton, C. et al. (2016). Where’s WALY?:: A proof of concept study of the ‘wellbeing adjusted life year’ using secondary analysis of cross-sectional survey data. Health Qual Life Outcomes, 14(126). https://doi.org/10.1186/s12955-016-0532-5 
5. MacAskill, W. (2022). What we owe the future. Basic books.
6. McGuire, J., Kaiser, C., & Bach-Mortensen, A. M. (2022). A systematic review and meta-analysis of the impact of cash transfers on subjective well-being and mental health in low-and middle-income countries. Nature Human Behaviour, 6(3), 359-370.
7. Millett P, & Snyder-Beattie A. (2017). Existential Risk and Cost-Effective Biosecurity. Health Security, 15(4):373-383. doi: 10.1089/hs.2017.0028.
8. Ord, T. (2019). The moral imperative towards cost-effectiveness. In H. Greaves and T. Pummer (Eds). Effective Altruism: Philosophical issues. Oxford University Press. 
9. Robinson, R. (1993). Cost-effectiveness analysis. British Medical Journal, 307(6907), 793-795. 
10. Wildeford, P. (2018). What is the cost-effectiveness of researching vaccines? https://forum.effectivealtruism.org/posts/3Tvu55ETMNx5T5tJ3/what-is-the-cost-effectiveness-of-researching-vaccines