In some recent work (particularly this article) I built models for estimating the cost effectiveness of work on problems when we don’t know how hard those problems are. The estimates they produce aren’t perfect, but they can get us started where it’s otherwise hard to make comparisons.
Now I want to know: what can we use this technique on? I have a couple of applications I am working on, but I’m keen to see what estimates other people produce.
There are complicated versions of the model which account for more factors, but we can start with a simple version. This is a tool for initial Fermi calculations: it’s relatively easy to use but should get us around the right order of magnitude. That can be very useful, and we can build more detailed models for the most promising opportunities.
The model is given by:
This expresses the expected benefit of adding another unit of resources to solving the problem. You can denominate the resources in dollars, researcher-years, or another convenient unit. To use this formula we need to estimate four variables:
-
R(0) denotes the current resources going towards the problem each year. Whatever units you measure R(0) in, those are the units we’ll get an estimate for the benefit of. So if R(0) is measured in researcher-years, the formula will tell us the expected benefit of adding a researcher year.
-
You want to count all of the resources going towards the problem. That includes the labour of those who work on it in their spare time, and some weighting for the talent of the people working in the area (if you doubled the budget going to an area, you couldn’t get twice as many people who are just as good; ideally we’d use an elasticity here).
-
Some resources may be aimed at something other than your problem, but be tangentially useful. We should count some fraction of those, according to how much resources devoted entirely to the problem they seem equivalent to.
-
-
B is the annual benefit that we’d get from a solution to the problem. You can measure this in its own units, but whatever you use here will be the units of value that come out in the cost-effectiveness estimate.
-
p and y/z are parameters that we will estimate together. p is the probability of getting a solution by the time y resources have been dedicated to the problem, if z resources have been dedicated so far. Note that we only need the ratio y/z, so we can estimate this directly.
-
Although y/z is hard to estimate, we will take a (natural) logarithm of it, so don’t worry too much about making this term precise.
-
I think it will often be best to use middling values of p, perhaps between 0.2 and 0.8.
-
And that’s it.
Example: How valuable is extra research into nuclear fusion? Assume:
-
R(0) = $5 billion (after a quick google turns up $1.5B for current spending, and adjusting upwards to account for non-financial inputs);
-
B = $1000 billion (guesswork, a bit over 1% of the world economy; a fraction of the current energy sector);
-
There’s a 50% chance of success (p = 0.5) by the time we’ve spent 100 times as many resources as today (log(y/z) = log(100) = 4.6).
Putting these together would give an expected societal benefit of (0.5*$1000B)/(5B*4.6) = $22 for every dollar spent. This is high enough to suggest that we may be significantly under-investing in fusion, and that a more careful calculation (with better-researched numbers!) might be justified.
Caveats
To get the simple formula, the model made a number of assumptions. Since we’re just using it to get rough numbers, it’s okay if we don’t fit these assumptions exactly, but if they’re totally off then the model may be inappropriate. One restriction in particular I’d want to bear in mind:
-
It should be plausible that we could solve the problem in the next decade or two.
It’s okay if this is unlikely, but I’d want to change the model if I were estimating the value of e.g. trying to colonise the stars.
Request for applications
So -- what would you like to apply this method to? What answers do you get?
To help structure the comment thread, I suggest attempting only one problem in each comment. Include the value of p, and the units of R(0) and units of B that you’d like to use. Then you can give your estimates for R(0), B, and y/z as a comment reply, and so can anyone else who wants to give estimates for the same thing.
I’ve also set up a google spreadsheet where we can enter estimates for the questions people propose. For the time being anyone can edit this.
Have fun!
A research area with a great deal of uncertainty but potentially high payoff is anti-ageing medicine. But how good is it to put more resources into?
To be concrete, let's look at the problem of being able to stop a majority of the ageing processes in cells. Let's:
So the estimate for y/z should be how many times historical efforts to solve the problem we'll need before there's a 20% total chance of success.
I think this is a particularly uncertain problem in various ways: our error bars on estimates are likely to be large, and the model is not a perfect fit. But it's also a good example of how we might begin with really no idea about how cost-effective we should think it is, and so produce a first number which can be helpful.
My estimates.
R(0): The SENS Foundation has an annual budget of around $4m, plus extra resources in the form of labour. Stem cell research has a global annual budget probably in the low billions, although it's not all directly relevant. Some basic science may be of relevance, but this is likely to be fairly tangential. Overall I will estimate $1b here, although this could be out by an order of magnitude in either direction.
B: Around 100m people die every year. It's unclear exactly what the effects of success would be on this figure, but providing a quarter ... (read more)