Wow, I didn't realize that evaluation existed! Thanks for sharing! (Though given that this evaluation only covers ~2 dozen grants for one fund, I think my overall assessment that there's little in the way of post-grant evaluation still holds).

Self-assessment via a simple google form is an interesting idea. My initial reaction is that it would be hard to structure the incentives well enough for me to trust self-assessments. But it still could be better than nothing. I'd be more excited about third party evaluations (like the one you shared) even if they were extremely cursory (e.g. flagging which ones have evidence that the project was even executed vs. those that don't) and selective (e.g. ignoring small grants to save time/effort).

I'd be a bit surprised if there wasn't already a process in place for retrospective analysis of this sort. Is there any public info available about if/how EA Funds do this?

When I looked at this as part of the 2022 red teaming contest, I found that “EA Funds has received roughly $50 million in donations and has made hundreds of grants, but has never published any post-grant assessments.” I’m almost positive there haven’t been any retrospective analyses of EA Funds grants since then.

This problem isn’t unique to EA Funds. I also found that EA Grants and the Community Building Grants program both lacked any kind of public post grant assessment. 

The unfortunate result of this situation is that while lots of time and money have been invested in various grantmaking programs, we don’t really know much about what types of grantmaking are most effective (e.g. granting to individuals vs. established organizations). It’s true that post-grant assessment is costly to conduct, but it’s disappointing that we haven’t made this investment which could significantly improve the efficacy of future grantmaking.

Yeah, fully agree with this. I hope now that EV and/or EV-affiliated people are talking more about this matter that they'll be willing to share what specific due diligence was done before accepting SBF's gifts and what their due diligence policies look like more generally. 

There have been various calls for an independent investigation into the relationships between EA and SBF. Do you think such an investigation is warranted? Why or why not?

EV should have due diligence processes in place, instigated by EA's first encounter with a disgraced crypto billionaire/major EA donor (Ben Delo). 

In February 2021, CEA (the EV rebrand hadn't happened yet) wrote: 

Here’s an update from CEA's operations team, which has been working on updating our practices for handling donations. This also applies to other organizations that are legally within CEA (80,000 Hours, Giving What We Can, Forethought Foundation, and EA Funds).

• “We are working with our lawyers to devise and implement an overarching policy for due diligence on all of our donors and donations going forward.
• We've engaged a third party who now conducts KYC (know your client) due diligence research on all major donors (>$20K a year).
• We have established a working relationship with TRM who conduct compliance and back-tracing for all crypto donations.

Thanks for writing up these thoughts Will, it is great to see you weighing in on these topics.

I’m unclear on one point (related to Elizabeth’s comments) around what you heard from former Alameda employees when you were initially learning about the dispute. Did you hear any concerns specifically about Sam’s unethical behavior, and if so, did these concerns constitute a nontrivial share of the total concerns you heard? 

I ask because in this comment and on Spencer’s podcast (at ~00:13:32), you characterize the concerns you heard about almost identically. In both cases, you mention a bunch of specific concerns you had heard (company was losing money, Sam’s too risky, he’s a bad manager, he wanted to double down rather than accept a lower return), but they all relate to Sam’s business acumen/competence and there’s no mention of ethical issues. So I’m hoping you can clarify why there’s a discrepancy with Time’s reporting, which specifically mentions that ethical concerns were a significant point of emphasis and that these were communicated directly to you:

[Alameda co-founders wrote a document that] “accuses Bankman-Fried of dismissing calls for stronger accounting and inflating the expected value of adding new exchanges, and said a majority of employees thought he was “negligent” and “unethical.” It also alleges he was “misreporting numbers” and “failing to update investors on poor performance.” The team “didn’t trust Sam to be in investor meetings alone,” colleagues wrote. “Sam will lie, and distort the truth for his own gain,” the document says.

…Mac Aulay and others warned MacAskill, Beckstead and Karnofsky about her co-founder’s alleged duplicity and unscrupulous business ethics, according to four people with knowledge of those discussions. Mac Aulay specifically flagged her concerns about Bankman-Fried’s honesty and trustworthiness, his maneuvering to control 100% of the company despite promising otherwise, his pattern of unethical behavior, and his inappropriate relationships with subordinates, sources say.

Excellent points, everything you write here makes a lot of sense to me. I really hope you’re able to find funding for the proposal to research the EA brand relative to other alternatives. That seems like a really fundamental issue to understand, and your proposed study could provide a lot of valuable information for a very modest price. 

Thanks so much for this additional data and analysis! Really interesting stuff here. To me, the most interesting things are:

1. The midpoint of the satisfaction scale being a pretty good threshold for the point at which we see behavior changes 
2. The relatively high frequency of people with high satisfaction temporarily stopping promoting EA (and the general flatness of this curve)
3. I was surprised that for the cohort that changed their behavior, “scandal” was just one of many reasons for dissatisfaction and didn’t really stand out. The data you provide looks quite consistent with Luke Freeman’s observation: “My impression is that that there was a confluence of things that peaking around the FTX-collapse. There was building hostility towards some of the more avant garde EA ideas an actions of those associated with EA^[1] (towards both accurate and also misrepresentations of those ideas and actions) that seemed to get traction around just prior to the launch of WWOTF which meant there were a lot of people/opinions that got a lot more light when WWOTF was getting lots of attention and FTX failed so spectacularly. Then there was so much energy and angst in the system (both within the community and its critics) that I think the other issues compounded more than any individual one would have. The confluence of all this has sadly left a bad taste in a lot of people's mouths that costs fairly uncontroversially good things a lot in terms of action and advocacy.”

That makes sense. That said, while it might not be possible to quantify the extent of selectin bias at play, I do think the combination of a) favoring simpler explanations and b) the pattern I observed in the data makes a pretty compelling case that dissatisfied people being less likely to take the survey is probably much more of an issue than dissatisfied people being more likely to take the survey to voice their dissatisfaction.

As I mentioned in point 3 of this comment:

Looking at the distribution of satisfaction scores in late 2022 vs late 2023, we see the reduction in satisfaction over that period coming from fewer people giving high ratings of 9 or 10, and more people giving low ratings of 3-5. But for the lowest ratings, we see basically no change in the number of people giving a rating of 2, and fewer people (almost nobody) now giving a rating of 1.

This suggests we could crudely estimate the selection effects of people dropping out of the community and therefore not answering the survey by assuming that there was a similar increase in scores of 1 and 2 as there was for scores of 3-5. My guess is that this would still understate the selection bias (because I’d guess we’re also missing people who would have given ratings in the 3-5 range), but it would at least be a start. I think it would be fair to assume that people who would have given satisfaction ratings of 1 or 2 but didn’t bother to complete the survey are probably also undercounted in the various measures of behavioral change.