(I only skimmed your post, and it has been some time since I've read either the GiveWell intervention reports or the studies they draw from)

I appreciate attempts to criticize/red-team existing EA organizations and EA evaluations of interventions. That said, this argument mostly falls flat for me.

My understanding is that the structure of the GiveWell recommendation for the Against Malaria Foundation (AMF) is really quite simple:

1. At the intervention level, there are strong, RCT-backed evidence that long-lasting insecticidal bednets are very good at preventing mosquito borne illnesses and overall decreasing child mortality.
2. At the charity level, AMF is unusually good at distributing such bednets at scale.

These arguments are not iron-clad. For example, for #1, maybe you think insecticidal bednets are so a priori implausible as an anti-malaria intervention that you would not trust any level of RCT evidence? But this just falls flat to me, as "bednets that prevent/kill mosquitoes makes it harder for malarial mosquitoes to sting kids at night" passes some very simple sanity checks, at least for me. (Or perhaps you think drawing GiveWell's conclusion from the RCTs is statistically wrong, because of reasons? If so, it'd be good to list the reasons!)

Another reason you might doubt #2 is relevant is if you're suspicious that AMF can confer similar results as would be implied by the RCTs.  For example, if you think the places AMF works in is so "out-of-distribution" relative to the RCTs, because of lower malarial load[1]. But my understanding is that a) the GiveWell analysis accounts for this and b) the malarial loads aren't that different.

There are a number of other reasons that I would not go into that engages with the argument structure. 

However, your critique does not engage with the structure of the argument, and instead[2] argues that because there's no direct empirical evidence of AMF's specific distribution of bednets  saving lives, we cannot assume that AMF's bednets save lives.

 I currently think your post is an overly myopic treatment of the evidence. For a better extension by my lights, I'd be interested to see more engagement from you on whether the structure of the original argument is wrong, or alternatively, why you think your alternative formulation/framework of the problem ought to be the preferred one. I would also be interested in a very different critique of AMF that takes GiveWell's structure as a given but argues that by those lights, AMF is not a good donation target (eg because the intervention research is actually shoddy, or because AMF is actually bad at delivering bednets).

[1] My understanding is that, in contrast, substantially lower worm load is a serious reason to be skeptical of the present-day impact of deworming interventions.

[2] You also argue that there's observational data against AMF's effectiveness because the countries they work in don't have obviously lower malarial loads. However I think causality is just pretty hard to determine from observational data, for reasons Charles mentions here.

In your post, I think your concerns are in two categories:

Issue A. Not tracking the effects of recipients (or more likely, initially trying to track but not finding no positive statistical effects and dropping data collection).

Indeed, AMF had a plan to monitor malaria case rates before and after distributions to prove their effectiveness.  However, when they actually collected the data they concluded the data was of poor quality and so abandoned this plan...I find this very worrying.  Maybe the data was of poor quality, but that is a reason for working harder in this area rather than abandoning it altogether.  In general, if we only have poor quality data about malaria in a region, doesn’t that mean we do not know how effective a bednet distribution will be?

Issue B. At the country level (not monitoring recipients of AMF nets but malaria levels in countries), there is no/limited/mixed evidence for malaria reduction:

Taking a step back from the Against Malaria Foundation to look at the malaria problem more generally, there is mixed evidence that bed net distributions reduce malaria case rates.  GiveWell has a macro review of the evidence which shows at the nation-level you cannot demonstrate any impact from all malaria control initiatives.  

...Malaria rates in Benin, DRC, Ghana, Mali & Sierra Leone increased as net coverage increased, which is more evidence that the malaria data being used is not great.  In central Africa malaria was trending downwards before bednet coverage was scaled up, further muddying the waters when trying to measure impact.

“Available data and studies appear to show some cases of apparent malaria control success, and also seem to indicate that the overall burden of malaria in Africa is more likely to be falling than rising. However, in most cases it is difficult to link changes in the burden of malaria to particular malaria control measures, or to malaria control in general, and the data remains quite limited and incomplete, such that we cannot confidently say that the burden of malaria has been falling on average.”

What you wrote and the reasoning is a complete and well reasoned line of thought from careful study of the AMF website. 

However, this is not sufficient evidence for strong updates against AMF. 

For me, it's not even enough evidence that would cause me to investigate this issue further.

The root issue/crux is that the "causal inference"/"causal identification" or the information you can get from statistics you collected here is very low, and far from a model of impact or finding the Truth.

Some perspectives:

Issue A: For the first issue, where tracking recipients was ineffective (or as you suggest and I would also find plausible, they found no statistical effect and then data collection was dropped), I don't know more than what you wrote, but finding no effects is plausible, even common, in highly successful interventions.

• The statistical power may be very low. To get intuition for this, remember that a life saved costs $5000 in expectation and a bednet costs ~$2. In some real statistical sense, you literally need thousands of bednets to get an "observation" of a death or life saved. So you may need many, hundreds of thousands, or really millions of bednets to get enough observations for statistical power. But that's just one layer of the difficulty and assumes perfectly balanced groups of treatment/control, demographics—you may need an order of magnitude more observations to do a proper observational study. Even generously, that's a large fraction of all the bednets distributed in a year. From this problem alone, my prior would be to find no effect and also I would expect it to impose large operational costs that many donors would find unacceptable (I would).
• The above implies a pretty clean, controlled test environment. E.g. two villages, one with bednets or one without, or really, two children in the same household, where one gets a treatment with one bednet and one without. This isn't going to happen in the actual program and the effects are wildly different if not controlled.
• Examples of random stories that's going to mess up inference: a principled bednet distributor might give nets to poorer families, families that have sicker children and adults. Since everyone probably knows bednets are effective, wealthier families might get their own (which is good, AMF can give to the really poor), and these wealthy families might get more premium bednets and treatments (e.g. $10 instead of $2), so you don't have a comparison group.
• There's even more pathological stories that mess up your inference: if you were a skilled implementor, working in this program on the group for many years, and you know you only have 100 bednets for 1000 people (maybe because the EAs got captured by the AI/futurist memes which diverted all the billionaire funds), it's possible that you know, working on the ground, who gets the bednets is very important, like by a factor of 2 or 4. That is, if you give the right bednets to the right people you can increase cost effectiveness by 200-400%. By definition, this skill isn't legible by some survey. So your very skill in giving bednets to the worst families, more afflicted by malaria, means that someone looking at the data will go "hey when we collect data for recipients of malaria nets, these families don't look better worse off, let's cancel this."

Issue B: Cross country effects

• The cross country sort of examinations suffers from all of the issues above, but is even weaker. For example, climate trends, poverty, institutional change are all going forces that will mess up results, and even this is description is a crude gesture at the realities of what is going on. What about other ways malaria can be contracted, besides sleeping in an bednet eligible bed?
• These confounding effects mean that nation studies might never find an effect at all, even with very effective interventions. One new major crux is how much coverage of bednets there is in a country. Again, I don't know anything about this more than reading your post, but if bednet distribution is 10% or even 30%, that is may not be enough to find an effect even if bednets were 100% effective.
• That's assuming that bednets were 100% effective. If bednets were even 1% effective (which by the way still makes them completely worth it and is consistent with the CEA of $5000 per life for ~$2 bednet), you may never be able to find an effect from an observational study.

Basically, cross country regressions aren't good without being embedded with a strong model/context and this domain is sort of an "also ran" in economics. 

Again, what you wrote and the reasoning is a complete and well reasoned line of thought from careful study of the AMF website. 

You said:

we may be ignoring evidence that the world is more complex than we thought, something which effective altruists ignore at their peril.

Like, to be clear, let's flip the evidence another way around: 

Imagine someone who came to you for money for a new project or new business. This person didn't understand the intervention, didn't understand the country or people. All they present is an argument they read from papers, with just country level observational data, or data from someone who they didn't know, who collected some data giving nets to families. 

If you were being asked to give money to this person, this information is not enough to trust them, (and it may even be wise to distrust them if this was the only argument they were able to present.)

Have you read this GiveWell page on bed nets? They state:

There is strong evidence that when large numbers of people use LLINs to protect themselves while sleeping, the burden of malaria can be reduced, resulting in a reduction in child mortality among other benefits.

Or this Cochrane review? 

Insecticide‐treated nets reduce child mortality from all causes by 17% compared to no nets (rate ratio 0.83, 95% CI 0.77 to 0.89; 5 trials, 200,833 participants, high‐certainty evidence). This corresponds to a saving of 5.6 lives (95% CI 3.6 to 7.6) each year for every 1000 children protected with ITNs. Insecticide‐treated nets also reduce the incidence of uncomplicated episodes of Plasmodium falciparum malaria by almost a half (rate ratio 0.55, 95% CI 0.48 to 0.64; 5 trials, 35,551 participants, high‐certainty evidence) and probably reduce the incidence of uncomplicated episodes of Plasmodium vivax malaria (risk ratio (RR) 0.61, 95% CI 0.48 to 0.77; 2 trials, 10,967 participants, moderate‐certainty evidence).

If the nation-level data isn't supportive of this, then perhaps this is worthy of further investigation to understand why it may be different from the trials. 

You seem to acknowledge this by saying 'Maybe the RCT evidence is so convincing that the noise of country-level data doesn’t matter' - but if your claim is that there is 'no evidence of impact' specifically at the country-level, then I'd encourage you to be clear about this with your heading. The statement that 'when you try to measure outputs there is no evidence of impact'  doesn't seem true.

I really think you ought to consider renaming this post given you've not even emailed GiveWell yet? It seems like you're planning to post a series on "concerns with AMF" before asking any follow up questions to anyone. Probably about 1000 people will see the title. There's some chance you could convince someone to stop donating to AMF just from the title - that tends to be how brains work, even though it isn't very rational.