I think the challenges inherent in evaluating animal charities go beyond even the difficult questions of animal sentience. Unlike in the case of direct health interventions, the impact of almost all of the activities by animal charities depends on extremely complicated facts about human psychology. Asking "how likely is leafletting to change behaviour," is more akin to asking "how likely is leafletting to change someone from a Republican to a Democrat" than it is to asking "how likely are bed nets to prevent malaria?"

Just a few of the complications of predicting long-term behaviour change through specific interventions:

-Unlike certain health risks which occur in specific types of situations, human psychology is extremely complicated and could in principle be influenced by events/conversations/memory recollections taking place at almost any moment throughout one's life.
-There are individual differences in how likely people are to be persuaded by particular tactics. So if only 2.5% of the population are even in principle persuadable by leaflets, and you happen to reach those 2.5% in an initial campaign (and, with perfect measurement, record the change), the prediction that a similar future campaign would cause change in 2.5% of the population would be completely wrong (because the 2.5% of reachable people would have already been reached). -Unlike in the case of health interventions, changing behaviour in relation to animals is taking place in direct opposition to a hugely powerful set of industries that are themselves spending millions of dollars trying to produce the exact opposite behaviour changes. So even if we discovered the holy grail intervention X that caused 100% of people to go vegan, Tyson foods would be busily working on messaging to convince people to eat more meat.

I'm not sure what the upshot of all of this is, but it does seem to me that influencing change in relation to animals is much more like running a political campaign. And people have been studying political tactics for thousands of years without developing any type of infallible playbook for winning. So while I think measuring animal charity effectiveness is extremely important and interesting, it also seems like it's a mistake to think that it could ever come close to reaching the type of reliability seen in other types of interventions.

The problem with HN's article wasn't just that it was "impolite" but that it mixed in a number of unfounded ad hominem attacks along with the more serious criticisms, arguments that implied people at ACE and other organisations were deliberately acting in bad faith. It seems to me that the proper way to respond to a mix of good arguments and bad arguments is to take the good arguments into account while dismissing the bad ones, and that seems to be what happened.

And just as a broader point, if someone regularly mixes ad hominem attacks with more serious points, it's not a good idea to signal boost the mixed arguments. If I say, "so-and-so is a liar and a thief who only cares about self-promotion, and also so-and-so incorrectly reported a particular study," the best response is to take the valid point about the study into consideration without promoting the argument as a whole.

Moreover, the methodological arguments put forward weren't entirely new; HL Lab director Harish Sethu's presentation at the EAA conference in Princeton, for example, had extremely detailed methodological criticisms that far surpassed the other criticisms I had seen to that point.

In other words, there's not a binary decision between ignoring the points altogether and praising the deeply flawed HN article. The good methodological points are being taken into consideration, as seen in some of the changes made and in this essay.