In this post, I present in general terms the simplified, cost-effectiveness estimation methodology that we plan to use in student projects that are evaluating a variety off-grid solar-electric technology access interventions in rural Africa.

Each of the interventions are focused on the Gobal Health and Welfare (GHW) cause area and either (A) Use funds contributed by philanthropic donors to subsidize access to a particular off-grid solar-electric product or service, or (B) Evaluate new, more efficient ways of administering philanthropic subsidies to potentially a large variety of distributed development activities.

Generally, the question that we want to answer is: "What is the probability that a particular intervention can be sufficiently cost-effective to compete with the most cost-effective GHW interventions currently supported by effective altruists (EAs)?"

Our focus is on calculating the MARGINAL cost-effectiveness (MCE) of an intervention under the key assumption that if the EA movement maximizes the MCE of donations for charities working in the GHW cause area, then this will maximize the good that EA donors can do for GHW given limited available resources. 

We define the marginal cost-effectiveness to be the inverse of the marginal cost per unit of impact (MCI) which is: 

              MCI =  Cd / NBs

Where Cd is the cost to donors per unit of intervention, and NBs is the Net benefit of the subsidized intervention that can be attributable to the impact of donor financing.

Because MCI and MCE are inversely related  minimizing MCI, maximizes MCE.

Note that when donations enable access to a new or higher quality product or service to become available for a population, this is solving a very common market failure. Typically access is enabled by a partial subsidy of a higher quality version of a product or service and without the subsidy the particular product or service may not be available at all in a particular market.  This can happen when there are "asymmetric information" market failures in a market, which are very common in lower income countries. See for example: "Market for Lemons" ...

When a market-for-lemons market failure occurs, low-benefit products "crowd out" higher quality, high-benefit products. Targeted subsidies lower the price of higher-quality products and the lower price for higher quality products allows higher-benefit products to enter the market and compete vis-a-vis cheaper, lower-benefit alternatives.

In this context, Cd in the equation for MCI represents the cost of subsidizing higher-quality products and services, while NBs represents the net increase in benefit of the higher quality product or service relative to the lower-quality alternative.

Target thresholds for MCI and MCE:

Different EA meta-charities define a variety of CE thresholds when they evaluate charities and interventions to determine of they are cost-effective.

Some of the different CE thresholds within the EA GHW community are as follows:

GiveWell:

•    ~$5000 per life saved 
•    ~$125/DALY, (Assuming an average of 40 DALY averted per life  saved)
•    6X to 10X as cost effective as cash transfers

Open Philanthropy:

•    $50/DALY
•     "increasing income for 4 people by ~1% for a year for $1" 

Center for Exploratory Altruism Research (CEARCH):

•      10X GiveWell CE of ~700 DALY/$100k
•       i.e ~$15/DALY

Details re: the Marginal Cost of Impact estimation

The two inputs to MCI, cost to donors (Cd) and the net benefit of a subsidized product (NBs), are conceptually simple quantities. But there are several factors that can influence their values in practice.  Here we discuss some of the details of estimating Cd and NBs in practice. 

Estimating Cd

Cd can be somewhat difficult to calculate for the subsidized distribution of products and services because the amount of subsidy versus the amount of customer payment can vary substantially.  The beneficiaries of a new product or service will of course want to get that product or service for a price that is as low as possible: preferably for free.  But that will not be the scenario in which the intervention is most cost-effective. So a key factor in estimating Cd is estimating the portion of product or service cost that is paid by donations.

In addition, there are administrative, management, and data collection costs associated with administering and implementing philanthropic subsidies. These costs will tend to be additional to the cost of the product or service subsidy.

One way of decomposing Cd into component factors is through the following equation:

    Cd = Cprod × Psub × Md 

Where Cprod is the cost of the product or service, Psub is the percent subsidy that is necessary to make the product affordable and desirable for the beneficiaries, and Md is the markup factor that represent the costs of administration and data collection for the subsidy donation.  For example, if for every $1 of subsidy, there are $0.30 of administrative, data collection and donation marketing costs, then Md = 1.3.

Estimating NBs

Similarly, estimating the net benefits of a new product or service is conceptually simple, but can be somewhat more complicated in practice.

The first key complication is that benefits need to be measured relative to what would have occurred without the subsidy program.  While a new and better product may provide clearly measurable benefits, what is harder to measure is what the product users might have done without the subsidized availability of the product.

The second factor that needs to be considered is that a higher-quality product may last many more years than a lower quality product. In this case, both the benefits and the costs of both the intervention and the no-intervention alternative need to be annualized so that they can be compared over a standard time interval of impact.  (For example see: https://www.epa.gov/sites/default/files/2017-09/documents/ee-0568-06.pdf for details of how to do this). The annual net benefit of the intervention NBann then becomes the difference between the annualized benefit of the intervention minus the annualized benefit of the alternative in case of no intervention. 

A third factor, is that even though a new product or service may provide a net benefit (such as pumping water for a garden, clean water, or household lighting), the beneficiaries may not utilize the full benefits. We characterize this with a utilization factor (UF) that is generally between 0 and 1, depending on how much the average program beneficiary utilizes the full benefits of a subsidized product or service.

And last but not least, the value of the annualized benefits may increase or decrease over time. Thus a discounted sum of a net value trend over the expected lifetime or duration of net benefits can be characterized with the net present value (i.e. discounted sum) of relative annual values. 

    NBs = [NBann × Fattrib × UF × NPVsum - (1-Psub) × Cprod]

In this equation, NBann is the annualized benefit of the intervention minus the benefit produced by the alternative without the intervention.  Fattrib represent the fraction of intervention adoption that can be attributed to donations that are supporting the intervention. UF is the fraction by which beneficiaries actually use the new product or service and actually reap its benefits.  And NPVsum is the discounted present value of relative annual benefit factors over the lifetime of the product or serve.  If the value of NBann is constant over time, then the relative annual benefit factors are all 1. If the value of the benefit increases 10%/year then [Annual Benefit Factor](year+1) = (1 + 10%) × [Annual Benefit Factor](year), etc. The term (1-Psub) × Cprod  represents the portion of the initial product price paid by the beneficiary.

And finally, NBs needs to be converted to standardized EA units such as DALYs, or people-percent-years of income increase in order to make the resulting CE estimates comparable to typical minimum CE donation criteria. Another useful cost-effectiveness metric that we like to use is "dollars of net beneficial impact per dollar donated."

Accounting for uncertainty and variability

Because the inputs that influence impact cost-effectiveness of an intervention can be both uncertain and variable, the results of a CE calculation is most appropriately provided as a probability distribution.

The standard approach to performing a benefit-cost calculation with variable or uncertain inputs is to perform a Monte Carlo simulation (The Wikipedia page on this topic is quite good: https://en.wikipedia.org/wiki/Monte_Carlo_method).  In our CE estimation with uncertain inputs, we implement a highly simplified Monte Carlo method that we call a simplified Monte Carlo or "poor man's" Monte Carlo calculation.

In our simplified Monte Carlo calculation, we initially estimate ranges for all or most of the input parameters, and represent these ranges by low, median, and high values. Given a probability distribution of what values a parameter may take, the low value represents the average value of the lowest 1/3 of possibilities, the median value represents the average of the middle 1/3 of probable values and the high value represents the average of the largest 1/3 of probably values. This approximates a probability distribution of possible input parameter values by three discrete values of equal probability. 

Once all of the input parameters are represented by three values of equal probability, then the CE result is calculated for all combinations of input parameters. If each of the input parameters are independent and uncorrelated, then the set of CE values that result from all combinations of inputs all have equal probability. A histogram of the full set of CE results is then constructed to illustrate the full range of possible CE values and their respective approximate probabilities. 

We illustrate this simplified Monte Carlo calculation below. 

Example calculations of CE distributions

Example calculation for long-lasting solar lights

To illustrate the CE estimation methodology above, we perform example calculations for nascent interventions that Solar4Africa.org implements in rural Malawi. One such intervention is the subsidized distribution of small solar lighting systems with cell phone charging to households in rural villages. The solar lighting system uses a newer, slightly more expensive battery chemistry (i.e., lithium titanate) which has cycle-life that is approximately ten times the cycle life of standard lithium ion batteries which are commonly used in solar lights in throughout rural Africa 

Here we calculate the MCE for subsidized long-lasting solar lights, using the equations specified above: 

         MCE = NBs/Cd 

      = [NBann×Fattrib×UF×NPVsum-(1-Psum)×Cprod]/[Cprod×Psub×Md]

The ranges of parameter values for the calculation are specified in Table 1. The system costs approximately $30 installed though depending on assembly and distribution efficiency, the installed cost can range from $20 to $40. Customers pay for the lighting system, but pay a subsidized price. The annual benefits of the solar lighting system consists of not having to pay for disposable batteries for a battery-powered light and not having to pay for cell phone charging at a local trading center.  These savings can range from $2/month to $5/month or $24 to $60 per year of savings. Impact attribution is less than 100% because it is possible for some households to buy alternative solar systems from other providers, and some households may not use the system for its entire lifetime. Because of the very long battery life, the system can last from 5 to 15 years so the discounted NPVsum ranges from 4.5 to 11 for a discount rate that may vary between 2%/year to 6%/year. 

Table 1: Input parameters for subsidized long-lasting solar lights cost-effectiveness

The results of the simplified Monte Carlo calculation is shown in Table 2 for the full set of 3⁷ = 2187 input parameter combinations. 

For this intervention, the most probable cost-effectiveness is approximately $8 of benefit per $1 donated. If these benefits accrue to people who have a per-capita income of $200/year, then this corresponds to the poverty-reduction cost-effectiveness that is equal to the threshold stated by Open Philanthropy of "increasing income for 4 people by ~1% for a year for $1." 

Table 2: Results of Monte Carlo calculation for subsidized long-lasting solar lights

MCI estimate for subsidized mosquito trap

A similar calculation can be performed for subsidizing solar powered mosquito traps to reduce the incidence of malaria in rural Malawi. Mass mosquito trapping is a malaria vector control method that has shown some promise, but which is not yet widely deployed. Solar4Africa.org has found that small DC fans can be used to make a very simple, low-power mosquito trap than can kill hundreds of mosquitos per night, even without using odor bait. 

Here we use our CE estimation methodology to estimate the marginal cost of impact of this intervention in units of $/DALY, assuming that the simple traps cost about $20 each, that the subsidy ranges from 50% to 100%, and that a baseline Malaria burden of disease of a family of 4 of 0.2 DALY/year can be reduced between 10% and 50%.

The equation for the marginal cost of impact is:

         MCI = Cd / NBs 

     = [Cprod×Psub×Md ] / [NBann×Fattrib×UF×NPVsum] 

Because the benefits are measured in units of DALY of disease impact avoided, we do not subtract the consumer cost of the mosquito trap. This is equivalent to saying that if the user is paying for a portion of the trap cost, then they will get other benefits from the trap such as greater comfort from fewer mosquito bites. 

Table 3 provides the input parameter values used in constructing a simplified Monte Carlo calculation of MCI. 

Table 3: Input parameters for subsidized solar-powered mosquito traps

Table 4: Results of Monte Carlo calculation of marginal cost of impact for subsidized malaria mosquito traps

Table 4 shows the distribution of results arising from the simplified Monte Carlo calculation of MCI.  About half of the time the intervention is not cost effective relative to the GiveWell threshold of $150/DALY averted.  Compared to the Open Philanthropy and CEARCH criteria of $50/DALY and $15/DALY, the intervention is not cost-effective most of the time. But we note that the intervention might be cost-effective relative to the more stringent criteria if the mosquito trapping device can be made to last a long time, if it can have a relatively large impact on vector populations, and if it is deployed in areas with a high burden of Malaria disease. 

Summary/Conclusion

In the Fall-23 EA student projects being conducted by Solar4Africa.org we are searching for new solar-powered interventions that can be cost-competitive with some of the most cost-effective GHW interventions that are supported by EAs. 

We have developed generic marginal cost-effectiveness equations and a simplified Monte Carlo technique which allows the calculations of a probability distribution of results for either marginal cost-effectiveness or the marginal cost of impact.  These methods allow the estimation of the probability that new interventions with uncertain inputs can reach different cost-effectiveness threshold criteria. 

Our hope is that this will aid in the development of new, highly cost-effective and a greater variety of easy-to-implement interventions that can help the EA movement produce greater GHW progress given the limited EA donation resources that are available now and in the near-term future.