Hi Pablo, important question! GFI decided to postpone re-evaluation to a future year to allow their teams more time to focus on new opportunities and challenges in the alternative proteins sector. Our researchers will be available to answer more questions about our 2024 charity recommendations in our AMA next week (Nov 19, 8-10am PT) on the FAST Forum. We hope to catch you there! Thanks, Holly

Hi Michael, I'm glad you're happy to see the cost-effectiveness models! And thank you for letting us know; there's been much more traffic than we're used which has made our website slow. We're actively trying to resolve this. Please hold on a bit while we get our site back up. Clearing your cache may also help :) Thanks for your patience everyone!

Thank you for spending time analyzing our methods. We appreciate those who are willing to engage with our work and help us improve the accuracy of our recommendations and reduce animal suffering as much as possible.

Based on previously received feedback and internal reflection, we have significantly updated our evaluation methods in the past year and will be publishing the details next Tuesday when we release our charity recommendations for 2024. From what we can tell from a quick skim, we think that our changes largely address Vetted Causes’ concerns here, as well as the detailed feedback we received last year from Giving What We Can (see also our response at the time) as part of their program that evaluates evaluators. Our cost-effectiveness analyses no longer use achievement or intervention scores, but rather directly calculate cost-effectiveness by dividing impact by cost, as you suggest. That being said, our work will never be perfect so we invite anyone reading this with the expertise to improve the rigor of our work to reach out, now or in the future.

Although your comments are related to methods that we no longer use, we’d like to spend more time understanding and engaging with them, learning from them, and potentially correcting any misconceptions. Unfortunately, we won’t have the opportunity to do so until after our charity recommendations are released next week. Additionally, it might be a comfort to know that for the past few months, Giving What We Can has been assessing ACE’s new evaluation methods along with a panel of other experts and that they intend to publish the results later this month.

Thank you.

- The ACE team

Rest assured, our new charity application process and commitment to serious impact-driven advocacy is no joke. We're here to keep the conversation on animal advocacy going strong, even on April Fool's Day!

Thank you, Toby! We appreciate the positive feedback and definitely share your thoughts about the value of this exercise.

- Max

Hi Michael, thanks a lot for the helpful comments, and for taking the time to be so thorough in your feedback. We've been thinking a lot about how to produce proxies for impact that can be meaningfully compared with one another, with BOTECs being one possible way to help achieve that, so it's really useful to get your views. We'll talk these through as a team as we consider improvements to our process for the coming years.

- Max

Thanks for the kind words! Really glad to hear you're likely to support the great work being done by our Recommended Charities.

Like you say, involvement with EA is a hard thing to judge: I wouldn't feel qualified to name directors that I believe are involved in EA, for example. Also, while all the charities we recommend use evidence-driven strategies to achieve the maximum benefits for animals, many of them might not consider themselves EA, or might not want to be labelled as such for strategic reasons. In answer to your more specific question, two of our Recommended Charities were incubated by Charity Entrepreneurship: Fish Welfare Initiative and Shrimp Welfare Project.

As it sounds like you're already aware, we don't assess 'EA alignment' as part of our evaluations. In part, this is because we think it's very likely that the animal advocacy movement needs to be pluralistic if it's going to bring about long-term systemic change, so we want to support a wide range of organizations rather than limiting ourselves to a specific sub-set. That said, we're also very aware of the risk of personal biases affecting our assessments and the need to mitigate that risk, which is one of the reasons that we seek to quantify our decision-making as much as possible and to be transparent about all of our methodology and decision-making. It might also be helpful to know that not all of our researchers identify as EA.

If this is something you're still concerned with, the best option for you might be to give to ACE's Recommended Charity Fund instead of to a single charity. Then your donation will be distributed among each of our recommended charities. That's also the easiest option if you want to simplify your giving while supporting the diversity of solutions that we believe are necessary to reduce animal suffering effectively. And donations to this fund will be matched if made by December 6!

I hope that's partway helpful despite not answering your specific question, and thanks again for engaging with our work and considering supporting our Recommended Charities' excellent work.

- Max

Thank you for your thoughtful question and interest in our evaluation approach. At ACE, we recognize the unique challenges present in our domain, where there is often less data and consensus on effective interventions compared to GiveWell's focus on global health and poverty. We also evaluate charities using a diverse range of 26 types of interventions, some with complex, long-term Theories of Change that are challenging to quantify.

For these reasons, we currently don't apply a specific cost-effectiveness bar across all charities, but we are consistently reevaluating this decision and exploring the potential of incorporating quantitative cost-effectiveness estimates. However, due to the diversity of interventions and the varying degrees of available data, applying a uniform cost-effectiveness bar to all charities, comparable to GiveWell's method, might not be feasible for us.

- Alina

First, we want to sincerely thank Giving What We Can for running this “Evaluating the Evaluators” exercise. We recognize that ACE has set ourselves a difficult task, compounded by the fact that we’re the only organization doing what we do. Therefore, receiving this kind of feedback is both very rare and very welcome. There’s a great deal in GWWC’s report that will help us improve our processes for 2024, which ultimately means more animals will be helped and spared. While we were disappointed that GWWC has decided not to defer to our recommendations this year or recommend our Movement Grants program as a top-rated fund, we were heartened by the positive points in their report and their optimism about ACE’s future, and look forward to receiving further helpful feedback in a future evaluation from them. We were also delighted that GWWC recommended the EA Animal Welfare Fund as an effective giving opportunity.

Second, as an organization that values transparency and seeks to be open about our own limitations, we appreciated GWWC’s same openness about the limitations to, and uncertainties around, their evaluation. As they noted, this included limitations to their animal welfare expertise, the early stage of the charity evaluation space in animal welfare, and the time constraints forcing them to take a minimum viable product approach to this evaluation. The bulk of this year’s process also coincided with the culmination of our charity recommendation decisions—which, combined with GWWC’s demanding deadlines, made some aspects of the process challenging for us. GWWC fully recognized this, and we are confident that any future evaluation exercise will be even more helpful than this year’s.  

Third, we were reassured that much of GWWC’s constructive feedback aligns with ACE’s own self-identified areas for improvement. For example, we agree that we need to continuously assess whether we want to give out fewer, larger Movement Grants than we do currently. We also agree that we should be more strategic in using the valuable feedback we get from our Movement Grants grantees to inform our own views on priority tactics and translate this into useful information for the broader animal advocacy movement.

We are also continuously working toward improvements to our charity evaluation methods, such as how to more sensitively capture differences in scope. As in previous years, we will be conducting a thorough review of the top-priority improvements to make to next year’s Movement Grants and Charity Evaluations programs. We will certainly draw on GWWC’s feedback for this while also acknowledging that capacity constraints will inevitably make it impossible to make all of the improvements we would like. 

Fourth, there are some elements of GWWC’s report that we did not fully agree with. For example, while we agree that there’s plenty of opportunity for improvements to our Cost Effectiveness model to ensure that it reflects the cost effectiveness of charities’ achievements as accurately as possible, we would like to highlight that this year’s model is the result of considerable research, external guidance, and exploration of alternatives. We built this year’s model systematically to try to capture the most important aspects of what makes achievements impactful, based on empirical evidence wherever possible, and the quantitative metric closest to impact on animals that we could access for all charities’ achievements (e.g., the number of people reached per dollar for an educational campaign). We consulted with several external experts on how to best combine these scores into a single score and went through several iterations to ensure that the scores held up in confidence checks. We also think it’s likely that GWWC is overestimating how easy it is to deliver on their recommendation of reliably estimating the “marginal cost effectiveness of a dollar spent on the charity, based on the charity’s specific context.” In some past rounds of our Charity Evaluations program, ACE carried out back-of-the-envelope-calculation (BOTEC)-type cost-effectiveness modeling using Guesstimate, which aligns with GWWC’s recommendation. However, ACE then decided to change this approach for the reasons outlined here. We continuously review this decision and are open to reintroducing elements of our past approach if we determine it would be valuable for more effectively advancing our theory of change. 

As another example of disagreement, GWWC noted that they would prefer our Movement Grants program to focus exclusively on national or international projects rather than the types of regional projects we have sometimes funded. However, given that one of the aims of the Movement Grants program is to build up the movement in priority regions with a relatively small animal advocacy movement, and that for some regions, we receive significantly more applications for regional projects than for high-quality, tractable, national-level applications, we expect that we will continue funding some regional projects that we consider particularly promising. Relatedly, GWWC disagrees with our view that funding projects in countries with very little animal advocacy representation should be a key part of ensuring the animal advocacy movement’s long-term success, noting that others may reasonably be more sympathetic to this view.

Fifth, we share GWWC’s commitment to prioritizing marginal funding to the projects where it will be the most cost effective. However, elements of ACE’s work—none of which are unique to us—make this particularly complex to achieve in practice. With our charity evaluations, for example, we only re-evaluate charities every two years, and we do not directly disburse most of the funding that we influence to our Recommended Charities. As such, ACE does not ​​regularly calculate cost effectiveness over a range of possible allocations and distributes funding only to those above a particular effectiveness bar; instead, we recommend a set of charities that we are confident will put additional funding toward effective use to help animals over a longer time horizon without our oversight. We make this decision based on all of our evaluation criteria, with a strong focus on Cost Effectiveness (which examines the effectiveness of past work) and Room For More Funding (which assesses whether a charity’s planned uses for funding over the next two years will be roughly similar to their past work). 

It is also worth noting that because we have shifted to one recommendation level for our charities this year (as opposed to the previous Top Charities/Standout Charities distinction), we plan to develop a new decision-making process that better accounts for the marginal cost effectiveness of funding disbursed from ACE’s Recommended Charity Fund. This will better enable us to leverage our grantmaking role in addition to our recommendation role.  

For our Movement Grants—especially smaller projects, projects benefiting species for whom few interventions have been tried, and projects in regions where the movement is particularly small, so there’s particularly little evidence—it is not currently possible to sufficiently investigate each project application to make meaningful cost-effectiveness estimates. We, therefore, rely on proxies such as the coherence of an applicant’s theory of change, the priority of their focus animal groups, and the neglectedness of the region in which they operate.

Additionally, because we believe that supporting a range of different approaches is essential for an effective animal advocacy ecosystem (and because our recommendations influence donor and public opinion), we want to feature a plurality of approaches with a strong potential for impact. Because of the high uncertainty about the most effective ways to help animals, and because the different interventions reinforce and facilitate each other, we think supporting a range of approaches is both necessary and beneficial for the animal advocacy ecosystem as a whole. We view this as preferable to diverting funding near-exclusively to charities and programs with the most convincing shorter-term theories of change, which runs the risk of dismissing potentially pivotal interventions due to measurability bias.

Sixth, we are glad that GWWC chose to recommend The Humane League (THL) based in part on our evaluation. As we note in our 2023 review, we view giving to THL as an excellent opportunity to support initiatives that create the most positive change for animals. At the same time, we are disappointed that GWWC chose to only recommend one of our Recommended Charities and to restrict grants for their corporate campaign work. After months of evaluation, we are confident that all of our Recommended Charities represent highly promising giving opportunities. We are also convinced of the need for a pluralistic and resilient movement incorporating a range of effective tactics toward different outcomes to achieve wellbeing for as many animals as soon as possible globally. 

We also want to make clear that while GWWC’s decision might imply that THL represents a superior giving opportunity compared to our other recommended charities, this is not a view that ACE shares. We view all of our recommended charities, including THL, as highly impactful giving opportunities. Following GWWC’s initial conclusion that they weren’t going to defer to our overall charity recommendations this year, we would have welcomed the opportunity to provide supporting materials for more than three of our Recommended Charities and to have had more time to do so.  

Lastly, and most importantly, we congratulate, once more, the latest additions to our list of Recommended Charities. We will diligently continue working to ensure that our approach to evaluation and our methods capture the full extent of charities’ work as accurately as possible, and we expect these improvements to be an ongoing endeavor that will continue for as long as ACE exists. At the same time, following months of research, preparation, and evaluation, we are confident that Kafessiz Türkiye, Dansk Vegetarisk Forening, Faunalytics, Fish Welfare Initiative, Good Food Institute, Legal Impact for Chickens, New Roots Institute, Shrimp Welfare Project, Sinergia Animal, The Humane League, and Wild Animal Initiative all do incredible work and represent extremely promising giving opportunities. We are pleased that GWWC’s report recognizes this, both through their direct recommendation of our Recommended Charity The Humane League, and through their strong recommendation to impact-maximizing donors to give to ACE’s recommended charities over the average animal welfare charity.

-ACE Team

Thank you for your comment! While there is limited hard evidence on the effectiveness of ballot measures, there have been some successful cases of them improving animal welfare standards over the last few decades in the U.S. (Schukraft, 2020). We agree that citizen initiatives such as ballot measures and other types of policy work are promising interventions, and hopefully, we will see more studies backing this up soon.

- Alina