"We don’t usually think of achievements in terms of what would have happened otherwise, but we should. What matters is not who does good but whether good is done; and the measure of how much good you achieve is the difference between what happens as a result of your actions and what would have happened anyway." - William MacAskill, Doing Good Better
Counterfactual thinking is fundamental to economic thinking, and this approach has been incorporated into Effective Altruism. The actual impact of your choices is based on what changed, not what happened. Per the forum wiki summary, "Counterfactual reasoning involves scenarios that will occur if an agent chooses a certain action, or that would have occurred if an agent had chosen an action they did not."
In this post, I'll argue that when counterfactual reasoning is applied the way Effective Altruist decisions and funding occurs in practice, there is a preventable anti-cooperative bias that is being created, and that this is making us as a movement less impactful than we could be.
Myopia and Hyperopia
To consider this, we need to revisit a fundamental assumption about how to think about impact, in light of the fact that individuals and groups can, should, and often do cooperate - both explicitly and implicitly. And to revisit the assumption, I want to note two failure modes for counterfactual reasoning.
First, myopic reasoning. Being a doctor saves lives, but arguably has little counterfactual value. And myopic reasoning can be far worse than this. As Benjamin Todd suggests imagining, "you’re at the scene of an accident and you see an injured person. In your enthusiasm to help, you push the paramedics out of the way and perform CPR on the injured person yourself. You’re successful and bring them back to consciousness, but because you’re less well-trained than the paramedics, you cause permanent damage to the person’s spine." Obviously, counterfactual thinking, appreciating that without your "help" the outcome will be better, can prevent this type of myopia.
But there is a second failure mode, which I'll call hyperopic reasoning. Myopia is nearsightedness, the inability to see things far away, and hyperopia is farsightedness, the inability to see things nearby. What does this look like in reasoning about outcomes, and how could it fail?
In December 2020, the United Kingdom and Sweden granted emergency approval for the Oxford–AstraZeneca vaccine, while Germany and the United States authorized the Pfizer-BioNTech vaccine, and the United States authorized the Moderna vaccine as well. While all of these were significantly better than the earliest vaccine, from Sinopharm in China, or the slightly later CoronaVac or Sputnik V, the counterfactual value of each vaccine seems comparatively miniscule because there were two other western vaccines that would have been approved without the development of this third option[1].
The Incidental Economist puts it clearly:
When you want to know the causal effect of an intervention (policy change, medical treatment, whatever) on something, you need to compare two states of the world: the world in which the intervention occurred and the world in which it did not. The latter is the counterfactual world... What we really want to know is how the world is different due to the intervention and only the intervention.
This is viewing the counterfactual from a single viewpoint. There is no shared credit for AstraZeneca, Pfizer, and Moderna, because each counterfactual is ignoring the nearby efforts, focused only on the global ones. But we certainly don't think that the value was zero - so what are we doing wrong?
We can try to resolve this without accounting for cooperation. Prior to the creation of the vaccines, it was unclear which would succeed. Each team has a prior potential to create a vaccine, and some chance that the others would fail. If each team has an uncorrelated 50% chance of succeeding, and with three teams, the probability of being the sole group with a vaccine is therefore 12.5% - justifying the investment. But that means the combined value of all three was only 37.5% of the value of success, and if we invested on that basis, we would underestimate the value for each. In this case, the value of a vaccine was plausibly in the hundreds of billions of dollars, so it's enough - but it's still wrong as a general rule.
Cooperation Within Effective Altruism, and Without
I think this problem is dealt with within the Effective Altruism movement partially by viewing it as a single actor[2]. Within Effective Altruism, people are remarkably cooperative, willing to forgo credit and avoid duplicating effort. If we're collectively doing two projects with similar goals, each of which may fail, we can view both as part of a strategy for ensuring the goals are achieved.
And there's a value in competition and multiple approaches to the same question, and this means some degree of internal competition is good. It's certainly useful that there is more than one funder, so that decisions are made independently, despite the costs imposed. It's probably good that there are two organizations doing career advising, instead of just having 80,000 Hours.
In business, this approach, with multiple firms engaged in related markets, Could be be called coopetition - cooperation on some things and competition on others. But the coopetition approach is imperfect. There is a natural tendency for people to seek credit for their ideas, and that often leads to suboptimal outcomes. (Which we can compensate for, at least partially.) Similarly, there is a cost to coordination, so even when people would like to cooperate, they may not find ways to do so. (But providing a list of ideas for others to pursue and helping launch projects designed to fill needs is useful.)
Where Effective Altruism more clearly falls down is external cooperation. Much of what Effective Altruism does is external-focused. (Biosecurity, Alternative protein, Global poverty reduction, and similar. Everything but community building, essentially.) Unfortunately, most of the world isn't allied with Effective Altruism, or is only implicitly cooperating. But most of the world at least generally shares its goals, if not its priorities. And that is unlikely to change in the near-term.
So who should get credit for what gets done? This matters - if we're not careful with our counterfactual reasoning, we could overcount or undercount our contribution, and make sub-optimal decisions.
Shapley Value
Lloyd S. Shapley gave us a fundamental tool for cooperation, the Shapley value, which is the (indisputably correct[3]) way to think about counterfactual value in scenarios with cooperation. If you don't know what it is, read or skim Nuño Sempere's post, and come back afterwards. But in short, in the same way that considering counterfactuals fixes myopia by considering what happens if you do nothing, Shapley values fix hyperopia by giving credit to everyone involved in cooperatively providing value.
And a large part of the value provided by Effective Altruism is only cooperative. We donate bed-nets, which get distributed by local partners, on the basis of data about malaria rates collected by health officials, when someone else is paying for the costs of the distribution, to individuals who need to do all the work of using the bed-nets. Then we walk away saying (hyperopically,) we saved a life for $5,000, ignoring every other part of the complex system enabling our donation to be effective. And that is not to say it's not an effective use of money! In fact, it's incredibly effective, even in Shapley-value terms. But we're over-allocating credit to ourselves.
The implicit question is whether we view the rest of the world as moral actors, or moral patients - and the latter is the direct cause of (the unfortunately named) White Savior Complex. This is the view that altruists decisions matter because they are the people responsible, and those they save are simply passive recipients. And (outside of certain types of animal welfare,) this view is clearly both wrong, and harmful.
"Non-EA Money"
I've used this phrase before, and heard others use it as well. The idea is that we'd generally prefer someone else spends money on something, because that leaves more money for us, counterfactually, to do good. But this is a anti-cooperative attitude, and when we have goals like pandemic prevention which are widely shared, the goal should be to maximize the good done, not the good that Effective Altruism does, or claims credit for. Sometimes, that means cooperating on use of funds, rather than maximizing counterfactual impact of our donation.
Does this matter? I think so, and not just in terms of potentially misallocating our funds across interventions. (Which by itself should worry us!) It also matters because it makes our decisions anti-cooperative. We're trying to leverage other spenders into spending money, rather than cooperating to maximize total good done.
For example, Givewell recently looked at likelihood of crowding out funding, with an explicitly adversarial model; if the Global Fund and/or PMI are providing less funding because of their donations, they don't view their own funding as counterfactual. And when we only take credit for counterfactual impact instead of Shapley impact, we're being hyperopic.
I'm unsure if there is a simple solution to this, since Shapley values require understanding not just your own strategy, but the strategy of others, which is information we don't have. I do think that it needs more explicit consideration - because as EA becomes more globally important, and coordination with other groups becomes increasingly valuable, playing the current implicitly anti-competitive strategy is going to be a very bad way to improve the world.
- ^
This example was inspired by Nuño Sempere's example of Leibniz and Newton both inventing calculus - which is arguably a far better example, because there was no advantage of having more manufacturing of the vaccine.
- ^
Despite the fact that it really, (really, really) isn't just one thing. Cooperation thankfully triumphs over even a remarkable degree of incoherence.
- ^
It is the uniquely determined way to allocate value across cooperating actors, in a mathematical sense, given decisions about how credit should be allocated like whether to allocate credit over groups or individuals, and whether to split by group, or by time spent, or some other metric.
Edited after more careful reading of the post
As you say in the post, I think all these things can be true:
1) The expected counterfactual value is all that matters (i.e. we can ignore Shapley values).
2) The 3 vaccine programs had zero counterfactual value in hindsight.
3) It was still the correct decision to work on each of them at the time, with the information that was available then.
At the time, none of the 3 programs knew that any of the others would succeed, so the expected value of each programme was very high. It's not clear to me why the '12.5%' figure in your probabilistic analysis is getting anything wrong.
If one vaccine program actually had known with certainty that the other two would already succeed, and if that really rendered their own counterfactual value 0 (probably not literally true in practice but granted for the sake of argument) then it seems very plausible to me that they probably should have focused on other things (despite what a Shapley value analysis might have told them).
It gets more complicated if you imagine that each of the three knew with certainty that the other two could succeed if they tried, but might not necessarily actually do it, because they will be reasoning similarly. Then it becomes a sort of game of chicken between the three of them as to which will actually do the work, and I think this is the kind of anti-cooperative nature of counterfactual value that you're alluding to. This is a potential problem with focusing only on counterfactuals, but focusing only on Shapley values has problems too, because it gives the wrong answer in cases where the decisions of others are already set in stone.
Toby Ord left a really good comment on the linked post on Shapley values that I think it's worth people reading, and I would echo his recommendation to read Parfit's Five Mistakes in Moral Mathematics for a really good discussion of these problems.
The most important thing about your decision theory is that it shouldn't predictably and in expectation leave you worse off than if you had used a different approach. My claim in the post is that we're using such an approach, and it leaves us predictably worse off in certain specific cases.
For example, I strongly disagree with the idea that it's coherent to say that all three programs would have zero value in hindsight, and that the true value is 12.5% each, because it means that in many plausible cases, where return on investment from a single working solution is, say, only 3x the bar for funding, we should fund none of them.
And regarding Toby's comment, I agree with him - the problem I pointed to in the last section is specifically and exactly relevant - we're committing to things on an ongoing and variable basis, along with others. It's a game-theoretic setup, and as he suggests, "Shapley [should be] applied when the other agents' decisions are still 'live'" - which is the case here. When EA was small, this was less problematic. We're big enough to factor into other large player's calculus now, so we can't pretend we move last in the game. (And even when we think we know we are, in fact, moving last in committing funds after everyone else, it is an iterated game, so we're not actually doing so.)
To arrive at the 12.5% value, you were assuming that you knew with certainty that the other two teams will try to create the vaccine without you (and that they each have a 50% chance of succeeding). And I still think that under that assumption, 12.5% is the correct figure.
If I understand your reasoning correctly for why you think this is incoherent, it's because:
If the 3 teams independently arrive at the 12.5% figure, and each use that to decide whether to proceed, then you might end up in a situation where none of them fund it, despite it being clearly worth it overall.
But in making this argument, you've changed the problem. The other 2 teams are now no longer funding the vaccine with certainty, they are also making decisions based on counterfactual cost-benefit. So 12.5% is no longer the right number.
To work out what the new right number is, you have to decide how likely you think it is that the other 2 teams will try to make a vaccine, and that might be tricky. Whatever arguments you think of, you might have to factor in whether the other 2 teams will be thinking similarly. But if you really do all have the same goals, and there's only 3 of you, there's a fairly easy solution here, which is to just talk to each other! As a group you can collectively figure out what set of actions distributed among the 3 of you will maximize the global good, and then just do those. Shapley values don't have to come into it.
It gets more complicated if there's too many actors involved to all get together and figure things out like this, or if you don't all have exactly the same goals, and maybe there is scope for concepts like Shapley values be useful in those cases. And you might well be right that EA is now often in situations like these.
Maybe we don't disagree much in that case. I just wanted to push back a bit against the way you presented Shapley values here (e.g. as the "indisputably correct way to think about counterfactual value in scenarios with cooperation"). Shapley values are not always the right way to approach these problems. For example, the two thought experiments at the beginning of Parfit's paper I linked above are specific cases where Shapley values would leave you predictably worse off (and all decision theories will have some cases where they leave you predictably worse off).
Let's make the problem as simple as possible; you have a simple intervention with 3 groups pursuing it. Each has an independent 50% chance of success, per superforecasters with an excellent track record, and talking about it and coordinating doesn't help, because each one is a different approach that can't benefit from coordination.
And I agree with you that there are cases where it's the wrong tool - but as you said, I think "EA is now often in situations like these," and we're not getting the answer right!
Edit: Vasco Grilo has pointed out a mistake in the final paragraph of this comment (see thread below), as I had misunderstood how to apply Shapley values, although I think the conclusion is not affected.
If the value of success is X, and the cost of each group pursuing the intervention is Y, then ideally we would want to pick N (the number of groups that will pursue the intervention) from the possible values 0,1,2 or 3, so as to maximize:
(1-(1/2)^N) X - N Y
i.e., to maximize expected value.
If all 3 groups have the same goals, they'll all agree what N is. If N is not 0 or 3, then the best thing for them to do is to get together and decide which of them will pursue the intervention, and which of them won't, in order to get the optimum N. They can base their decision of how to allocate the groups on secondary factors (or by chance if everything else really is equal). If they all have the same goals then there's no game theory here. They'll all be happy with this, and they'll all be maximizing their own individual counterfactual expected value by taking part in this coordination.
This is what I mean by coordination. The fact that their individual approaches are different is irrelevant to them benefiting from this form of coordination.
'Maximize Shapley value' will perform worse than this strategy. For example, suppose X is 8, Y is 2. The optimum value of N for expected value is then 2 (2 groups pursue intervention, 1 doesn't). But using Shapley values, I think you find that whatever N is, the Shapley value of your contribution is always >2. So whatever every other group is doing, each group should decide to take part, and we then end up at N=3, which is sub-optimal.
Hi Toby,
I do not think this is correct. If we consider a game with N players where each has to pay c to have a probability p of achieving a value of V:
In David's example, N = 3, and p = 0.5, so S = (7/24) V - c. For your values of V = 8, and c = 2, S = 1/3. This is not higher than 2.
From the formula for the Shapley value, maximising it is equivalent to maximising:
I have concluded in this Sheet the above is a strictly decreasing function of N (which tends to 0), so Shapley value is maximised for the smallest possible number of players. This makes intuitive sense, as there is less credit to be shared when N is smaller.
The smallest possible number of players is 1, in which case the Shapley value equals the counterfactual value. In reality, N is higher than 1 in expectation, because it can only be 1 or higher. So, since the Shapley value decreases with N, assuming a single player game will tend to overestimate the contribution of that single player. I think David was arguing for this.
In any case, I do not think it makes sense to frame the problem as deciding what is the best value for N. This is supposed to be the number of ("live") agents in the problem we are trying to solve, not something we can select.
Thank you for this correction, I think you're right! I had misunderstood how to apply Shapley values here, and I appreciate you taking the time to work through this in detail.
If I understand correctly now, the right way to apply Shapley values to this problem (with X=8, Y=2) is not to work with N (the number of players who end up contributing, which is unknown), but instead to work with N', the number of 'live' players who could contribute (known with certainty here, not something you can select), and then:
Have I understood the Shapley value approach correctly? If so, I think my final conclusion still stands (even if for the wrong reasons) that a Shapley value analysis will lead to sub-optimal N (number of players deciding to participate). Since the optimal N here is 2 (or 1, which has same value).
As for whether the framing of the problem makes sense, with N as something we can select, the point I was making was that in a lot of real-world situations, N might well be something we can select. If a group of people have the same goals, they can coordinate to choose N, and then you're not really in a game-theory situation at all. (This wasn't a central point to my original comment but was the point I was defending in the comment you're responding to)
Even if you don't all have exactly the same goals, or if there's a lot of actors, it seems like you'll often be able to benefit by communicating and coordinating, and then you'll be able to improve over the approach of everyone deciding independently according to a Shapley value estimate: e.g. Givewell recommending a funding allocation split between their top charities.
Since I was calculating the Shapley value relative to doing nothing, it being positive only means taking the action is better than doing nothing. In reality, there will be other options available, so I think agents will want to maximise their Shapley cost-effectiveness. For the previous situation, it would be:
For the previous values, this would be 7/6. Apparently not very high, considering donating 1 $ to GWWC leads to 6 $ of counterfactual effective donations as a lower bound (see here). However, the Shapley cost-effectiveness of GWWC would be lower than their counterfactual cost-effectiveness... In general, since there are barely any impact assessments using Shapley values, it is a little hard to tell whether a given value is good or bad.
In a single person game, or one where we're fully aligned and cooperating, we get to choose N. We should get to the point where we're actively cooperating, but it's not always that easy. And in a game-theoretic situation, where we're only in control of one party, we need a different approach than either saying we can choose where to invest last, when we can't, and I agree that it's more complex than Shapley values.
Hi Toby,
Regarding:
You are most likely aware of this, but I just wanted to clarify Shapley value is equal to counterfactual value when there is only 1 ("live") agent. So Shapley value would not leave us predictably worse off if we get the number of agents right.
Point taken, although I think this is analogous to saying: Counterfactual analysis will not leave us predictably worse off if we get the probabilities of others deciding to contribute right.
This isn't a problem with expected utility maximization (with counterfactuals), though, right? I think the use of counterfactuals is theoretically sound, but we may be incorrectly modelling counterfactuals.
It's a problem with using expected utility maximization in a game theoretic setup without paying attention to other players' decisions and responses - that is, using counterfactuals which don't account for other player actions, instead of Shapley values, which are a game theoretic solution to the multi-agent dilemma.
I'm torn with this post as while I agree with the overall spirit (that EAs can do better at cooperation and counterfactuals, be more prosocial), I think the post makes some strong claims/assumptions which I disagree with. I find it problematic that these assumptions are stated like they are facts.
First, EA may be better at "internal" cooperation than other groups, but cooperation is hard and internal EA cooperation is far from perfect.
Second, the idea that correctly assessed counterfactual impact is hyperopic. Nope, hyperopic assessments are just a sign of not getting your counterfactual right.
Third, the idea that Shapley values are the solution. I like Shapley values but only within the narrow constraints for which they are well specified. That is, environments where cooperation should inherently be possible: when all agents agree on the value that is being created. In general you need an approach that can hand both cooperative and adversial environments and everything in between. I'd call that general approach counterfactual impact. I see another commentor has noted Toby's old comments about this and I'll second that.
Finally, economists may do more counterfactual reasoning than other groups but that doesn't mean they have it all figured out. Ask your average economist to quickly model a counterfactual and it could easily end up being as myopic or hyperopic too. The solution is really to get all analysts better trained on heuristics for reasoning about counterfactuals in a way that is prosocial. To me that is what you get to if you try to implement philosophies like Toby's global consequentialism. But we need more practical work on things like this, not repetitive claims about Shapley values.
I'm writing quickly and hope this comes across in the right spirit. I do find the strong claims in this post frustrating to see, but I welcome that you raised the topic.
I'm frustrated by your claim that I make strong claims and assumptions, since it seems like what you disagree with me about are conclusions you'd have from skimming, rather than engaging, and being extremely uncharitable.
First, yes, cooperation is hard, and EAs do it "partially." I admit that fact, and it's certainly not the point of this post, so I don't think we disagree. Second, you're smuggling the entire argument into "correctly assessed counterfactual impact," and again, sure, I agree that if it's correct, it's not hyperopic - but correct requires a game theoretic approach, which we don't generally use in practice.
Third, I don't think we should just use Shapley values, which you seem to claim I believe. I said in the conclusion, "I'm unsure if there is a simple solution to this," and I agreed that it's relevant only to where we have goals that are amenable to cooperation. Unfortunately, as I pointed out, in exactly those potentially cooperative scenarios, it seems that EA organizations are the ones attempting to eke out marginal attributable impact instead of cooperating to maximize total good done. I've responded to the comment about Toby's claims, and again note that those comments are assuming we're not in a potentially cooperative scenario, or that we are pretending we get to ignore the way others respond to our decisions over time. And finally, I don't know where your attack on economists is coming from, but it seems completely unrelated to the post. Yes, we need more practical work on this, but more than that, we need to admit there is a problem, and stop using poorly reasoned counterfactuals about other group's behavior - something you seem to agree with in your comment.
I really like this, thanks!
Another point to perhaps add (not well formed thought) is that 2 groups may be doing the exact same thing with the exact same outcome (say 2 vaccine companies), but because they have such different funding sources and/or political influence there remains enormous counterfactual good.
For example in Covid, many countries for political reasons almost "had" to have their own vaccine so they could produce the vaccine themselves and garner trust in the population. I would argue that none of America, China and Russia would have freely accepted each other's vaccines, so they had to research and produce their own even if it didn't make economic sense. The counterfactual value was their not because the vaccine was "needed" in a perfect world, but because it was needed in the weird geopolitcal setup that happens to exist. If those countries hadn't invented and produced their own vaccines, there would have been huge resistance in importing one from another country. Even if it was allowed and promoted how many Americans would have accepted using sinovax?
OR 2 NGOs could do the same thing (e.g. giving out bednets), but have completely different sources of funding. One could be funded by USAID and the other by DIFID. It might be theoretically inefficient to have 2 NGOs doing the same thing, but in reality they do double the good and distribute twice as many nets because their sources of income don't overlap at all.
The world is complicated
I didn't express this so well but I hope you get the jist....
Yes, this is a reason that in practice, applying Shapley values will be very tricky - you need to account for lots of details. That's true of counterfactuals as well, but Shapley values make it even harder. (But given that we're talking about Givewell and similar orgs allocating tens of millions of dollars, the marginal gain seems obviously worth it to me.)
Thanks for the post!
One case I've previously thought about is that some naive forms of patient philanthropy could be like this - trying to take credit for spending on the "best" interventions.
I've polished a old draft and posted it as short-form with some discussion of this (in the When patient philanthropy is counterfactual section).
Thanks David, this made me think quite a bit.
This comment starts out nitpicky but hopefully gets better
"If each team has an uncorrelated 50% chance of succeeding, and with three teams, the probability of being the sole group with a vaccine is therefore 12.5% - justifying the investment. But that means the combined value of all three was only 37.5% of the value of success, and if we invested on that basis, we would underestimate the value for each"
There's something wrong with this calculation. I don't think it makes sense to sum marginal values in this way. It's like saying that if the last input has zero marginal value then the average input also has zero marginal value. But if we lower inputs based on this, the marginal value will go up and we will have a contradiction.
You want the expected value of trying to develop each vaccine, which requires an assumption about how others decide if to try or not. Then, it won't be obvious that all three try, and in the cases they don't , the marginal value of those that do is higher.
I think you're right that we should be careful with counterfactual thinking, but cooperative game theory won't solve this problem. Shapley value is based on the marginal contribution of each player. So it's also counterfactual.
Instead, I think the natural solution to the problem you outline is to think more clearly about our utility function. We sometimes talk as if it's equal to the sum of utilities of all present and future sentient beings and sometimes as if it's equal to the marginal contribution attributable to a specific action by EA. neither is good. The first is too broad and the latter too narrow. If we think EA is about doing good better, we should care about converting people and resources to our way of doing good and also about to be transparent and credible and rigorous about doing good. So if we cooperate with others in ways that promote these goals, great, otherwise, we should try to compete with them for resources.
Are you suggesting we maximize Shapley values instead of expected utility with counterfactuals? That's going to violate standard rationality axioms, and so (in idealized scenarios with "correctly identified" counterfactual distributions) is likely to lead to worse outcomes overall. It could recommend, both in practice and in theory, doing fully redundant work for just for the credit and no extra value. In the paramedic example, depending on how the numbers work out, couldn't it recommend perversely pushing the paramedics out of the way to do CPR yourself and injuring the individual's spine, even knowing ahead of time you will injure them? There's a coalition - just you - where your marginal/counterfactual impact is to save the person's life, and giving that positive weight could make up for the injury you actually cause.
I think we should only think of Shapley values as a way to assign credit. The ideal is still to maximize expected utility (or avoid stochastic dominance or whatever). Maybe we need to model counterfactuals with other agents better, but I don't find the examples you gave very persuasive.
In scenarios where we are actually cooperating, maximizing Shapley values is the game-theoretic optimal solution to maximize surplus generated by all parties - am I misunderstanding that? And since we're not interested in maximizing our impact, we're interested in improving the future, that matters more than maximizing what you get credit for.
So yes, obviously, if you're pretending you're the only funder, or stipulating that you get to invest last and no one will react in future years to your decisions, then yes, it "is likely to lead to worse outcomes overall." But we're not in that world, because we're not playing a single player game.
Do you mean maximizing the sum of Shapley values or just your own Shapley value? I had the latter in mind. I might be mistaken about the specific perverse examples even under that interpretation, since I'm not sure how Shapley values are meant to be used. Maximizing your own Shapley value seems to bring in a bunch of counterfactuals (i.e. your counterfactual contribution to each possible coalition) and weigh them ignoring propensities to cooperate/coordinate, though.
On the other hand, the sum of Shapley values is just the value (your utility?) of the "grand" coalition, i.e. everyone together. If you're just maximizing this, you don’t need to calculate any of the Shapley values first (and in general, you need to calculate the value of the grand coalition for each Shapley value). I think the point of Shapley values would just be for assigning credit (and anything downstream of that), not deciding on the acts for which credit will need to be distributed.
If you're maximizing this sum, what are the options you're maximizing over?
On one interpretation, if you're maximizing this only over your own actions and their consequences, including on others' responses (and possibly acausal influence), it's just maximizing expected utility.
On another interpretation, if you're maximizing it over everyone's actions or assuming everyone else is maximizing it (and so probably that everyone is sufficiently aligned), then that would be ideal (game-theoretically optimal?), but such an assumption is often unrealistic and making it can lead to worse outcomes. For example, our contributions to a charity primarily supported (funding, volunteering, work) by non-EAs with little room for more support might displace that non-EA support towards far less cost-effective uses or even harmful uses. And in that case, it can be better to look elsewhere more neglected by non-EAs. The assumption may be fine in some scenarios (e.g. enough alignment and competence), but it can also be accommodated in 1.
So, I think the ideal target is just doing 1 carefully, including accounting for your influence over other agents and possibly acausal influence in particular.
As I said in the post, "I'm unsure if there is a simple solution to this, since Shapley values require understanding not just your own strategy, but the strategy of others, which is information we don't have. I do think that it needs more explicit consideration..."
You're saying that "if you're maximizing this only over your own actions and their consequences, including on others' responses (and possibly acausal influence), it's just maximizing expected utility."
I think we agree, modulus the fact that we're operating in conditions where much of the information we need to "just" maximize utility is unavailable.
It seems like you're overselling Shapley values here, then, unless I've misunderstood. They won't help to decide which interventions to fund, except for indirect reasons (e.g. assigning credit and funding ex post, judging track record).
You wrote "Then we walk away saying (hyperopically,) we saved a life for $5,000, ignoring every other part of the complex system enabling our donation to be effective. And that is not to say it's not an effective use of money! In fact, it's incredibly effective, even in Shapley-value terms. But we're over-allocating credit to ourselves."
But if $5000 per life saved is the wrong number to use to compare interventions, Shapley values won't help (for the right reasons, anyway). The solution here is to just model counterfactuals better. If you're maximizing the sum of Shapley values, you're acknowledging we have to model counterfactuals better anyway, and the sum is just expected utility, so you don’t need the Shapley values in the first place. Either Shapley value cost-effectiveness is the same as the usual cost-effectiveness (my interpretation 1) and redundant, or it's a predictably suboptimal theoretical target (e.g. maximizing your own Shapley value only, as in Nuno's proposal, or as another option, my interpretation 2, which requires unrealistic counterfactual assumptions).
The solution to the non-EA money problem is also to just model counterfactuals better. For example, Charity Entrepreneurship has used estimates of the counterfactual cost-effectiveness of non-EA money raised by their incubated charities if the incubated charity doesn't raise it.
You're right that Shapley values are the wrong tool - thank you for engaging with me on that, and I have gone back and edited the post to reflect that!
I'm realizing as I research this that the problem is that act-utilitarianism fundamentally fails for cooperation, and there's a large literature on that fact[1] - I need to do much more research.
But "just model counterfactuals better" isn't a useful response. It's just saying "get the correct answer," which completely avoids the problem of how to cooperate and how to avoid the errors I was pointing at.
Kuflik, A. (1982). Utilitarianism and large-scale cooperation. Australasian Journal of Philosophy, 60(3), 224–237.
Regan, Donald H., 'Co-operative Utilitarianism Introduced', Utilitarianism and Co-operation (Oxford, 1980)
Williams, Evan G. "Introducing Recursive Consequentialism: A Modified Version of Cooperative Utilitarianism." The Philosophical Quarterly 67.269 (2017): 794-812.