Summary

We introduce Clear Solutions, a Charity Entrepreneurship (now AIM) incubated charity founded in September 2023. Our focus is the prevention of deaths of young children from diarrhoea, an illness that kills approximately 444,000 children under-5 every year.  

From December 2023 to February 2024, we ran a pilot distribution of low-cost, highly effective treatments for diarrhoea, oral rehydration solution and zinc (ORSZ) in Kano, Nigeria, with implementation partner iDevPro Africa. We estimate having reached ~6900 children under-5. The intervention, based upon a randomised controlled trial in Uganda (Wagner et al, 2019), provides free co-packaged ORS and zinc (“co-packs”) door-to-door to all households with children under 5 years old. The distribution is performed by local Community Health Workers (CHWs), who provide guidance and printed instructions on ORSZ usage during the visit.

We surveyed communities pre- and post-intervention, allowing 6 weeks between ORSZ distribution and follow-up surveys for diarrhoea cases to accumulate. At these survey rounds, we recorded the timing of the child’s last diarrhoea episode (if applicable) and how they were treated (if at all). Our primary outcome measure is the change in ORSZ usage rates pre-to-post intervention, though also we collected extensive contextual data to monitor operations and guide program improvements.

This post summarises our preliminary analysis and conclusions. A more detailed report is available on our website here. We were kindly supported by knowledgeable advisors, but did not have an academic partnership, nor has this analysis been peer-reviewed. Nonetheless, we believe there is value in sharing our results and learnings with this community.

Results in brief: Across 4 wards (geographic areas) of differing rurality, baseline usage rates for under-5s’ last diarrhoea episode in the preceding 4 weeks were reported at a range across wards of 44.7% - 50.9% for ORS and 11.1% - 26.7% for ORS+zinc when asked directly.  At follow-up post-intervention, the usage rate for the preceding 4-weeks was reported at a range across wards of 90.0 - 97.7% for ORS and 88.2% - 94.1% for ORSZ. (95% margins of error up to 10pp and are not shown here for readability; see Results for details.)

Superficially, this indicates a change of 42.0 - 52.8 percentage points (pp) in ORS use and 61.5 - 83.0pp for ORSZ. However, we treat this result with caution, with specific concerns such as social desirability bias in survey responses inflating true values. We discuss more in Limitations below.

Conclusions in brief: We consider this to be a solid result in favour of the intervention having a strong potential to prevent deaths in a cost-effective manner in the Nigerian context. (We do not estimate cost-effectiveness in this report, but will be working on a follow-up with that). There are, however, clear limitations in the pilot that warrant considerable down-weighting of our results, though we do not expect this to change the conclusions qualitatively.

Introducing Clear Solutions

Clear Solutions was founded in September 2023 with the support of Charity Entrepreneurship (now AIM). Our mission is to prevent deaths of young children from diarrhoea, a leading cause of death for under-5s globally, in a cost-effective and evidence-based manner.

The 1970s medical breakthrough, Oral Rehydration Solution (ORS), a dosed mixture of sugar, salts and water, unlocked the possibility of preventing >90% of diarrhoeal deaths at full coverage. The addition of zinc can reduce diarrhoea duration and recurrence, and the World Health Organisation recognised this in 2019 by adding co-packaged ORS and Zinc (together, ORSZ) to their model National Essential Medicines list.

The Global Health community has invested heavily in reducing diarrhoeal mortality with ORS for several decades, and significant progress has been made in many countries. For example, efforts have been made to improve the market dynamics for ORS and zinc, enable health workers, and increase demand from caregivers (Goh & Yobo, 2016). However, about 444,000 children under-5 still die from diarrhoea each year, with sub-Saharan Africa suffering a disproportionate burden.

A 2019 randomised controlled trial in Uganda provided inspiration for our approach as a new organisation in this historied domain. Wagner et al. 2019 found a >30% (19 percentage points) increase in ORSZ usage in communities where ORSZ was provided free door-to-door by Community Health Workers (CHWs), compared to a control of CHWs’ normal mode which included selling ORSZ. Importantly, the ORSZ was provided free and preemptively (ie, for caregivers to retain for immediate availability when a child suffers diarrhoea). Some earlier academic literature (eg. Zwisler et al, 2013, Digre et al, 2016) indicated caregivers were sceptical towards ORS, perceiving it as less like “real medicine” than injections or tablets.

Cost-effectiveness analyses by Charity Entrepreneurship, and later GiveWell in their August 2023 ORSZ Program report, modelled free pre-emptive community distribution. Givewell estimated $2.6K per death averted in Nigeria, with Charity Entrepreneurship ranging as low as $2.1K in Chad and $6.9K average over 8 high burden SSA countries.

Geographic selection and contextual learning

Diarrhoeal mortality and existing ORSZ usage rates are major determinants of our impact potential. 5-10 countries in sub-Saharan Africa have the combination of high mortality rates, low ORSZ usage, and large enough population to make sense for our first intervention. After initial exploration of implementing in Ethiopia, Madagascar and Nigeria, we ultimately decided to focus first on Nigeria for reasons of scale, language and partner networks.

Within Nigeria, we were again guided by mortality and ORSZ usage data at a state level, and applied more practical considerations, with an initial plan to execute ‘micropilots’ in up to 3 of our shortlisted states, Enugu, Federal Capital Territory, Gombe, and Kano. Our November 2023 visits involved meeting potential partner teams and Ministry of Health representatives to secure support for the intervention and plan for implementation.

During visits to Gombe and Kano, we performed focus groups and individual interviews with key stakeholders including caregivers, community leaders, community health workers, and facility health workers. These were valuable in understanding the local context: awareness of, and demand for, ORS was relatively high (contrary to our initial impression from some literature, but possibly as a result of subsequent global health initiatives), but the cost of the ORS itself was cited as significant barriers to regular use. Some participants also indicated that ORS distributed for free may be ‘diverted’ outside of our intended under-5s usage groups, with demand existing across the full range of age groups.

We were pleasantly surprised to find that the partner in Kano, iDevPro Africa, had a window to secure governmental approvals and execute very quickly at a larger scale during December 2023. Thus our plan pivoted to capitalise on this opportunity and we proceeded with the pilot as outlined below.

Pilot overview

Six months into this journey, we have preliminary results from our pilot of community distribution of co-packaged oral rehydration solution and zinc (ORSZ) diarrhoea treatments in Kano, Nigeria.

The pilot ran from December 2023 to February 2024, with ORSZ distributions in 20 communities across 4 wards in 2 Local Government Areas. We estimate having reached ~6900 children under-5, based on distribution and procurement records. Survey staff were deployed before, immediately after, and 6 weeks after distribution to gather operational learnings and enable evaluation of the intervention impact.

Community Health Workers (CHWs) from the National Primary Healthcare Development Agency CHIPS (Community Health Influencers, Promoters & Services) program - and Volunteer Community Mobilisers, one of its predecessors - performed the ORSZ distribution. The CHIPS agents and VCMs are a volunteer cadre within the overall primary healthcare system and are well known within their communities.

Over a period of 3-5 days, each CHW visited households in their community and, if they had children under-5 and a caregiver present, the CHW talked them through an ORSZ instructions sheet and gave one ORSZ co-pack (2 ORS sachets, 10 zinc tablets) per child under-5. Households were marked with chalk to help with later identification by survey staff. Paper distribution tracking sheets were used to monitor progress.

Survey staff visited a random sample of households at Baseline (before CHW visits), without prior knowledge of whether under-5s were present, and later used the CHIPS/VCMs’ chalk markings to identify eligible households for Spotcheck and Follow-Up surveys. A smartphone survey app (SurveyCTO Collect) was used to capture survey results.

Preliminary analysis and results

We performed the following analysis on operational data from the intervention and surveys. Given the absence of peer review or academic partnership, we present just summary-level highlights to avoid implying levels of precision or rigour beyond the practical reality. We calculated our 95% confidence intervals using z-scores (with combined variances for the difference in proportions calculations).

Operationally, surveys and distribution were divided into 2 stages by location, with work in Ward 1 (Urban Slum) preceding Wards 2-4 (Urban, Peri-Urban, Rural) by approximately 2 weeks to enable operational learnings to be incorporated.

Data collection was focussed on a primary outcome of change in ORSZ usage rate. We also collected other contextual data, inspired by comparable datasets such as the Demographic Health Survey (DHS) and our benchmark study, Wagner et al. 2019 in Uganda.

Diarrhoea Prevalence

One of the first data points we ascertained was the timing of the last episode of diarrhoea suffered by each child under 5 as reported by the caregiver. From this, we calculate the prevalence over defined periods preceding the survey.

Note: Diarrhoea prevalence measures the percentage of children under the age of 5 who had diarrhoea in the defined weeks preceding the survey. All margins of error shown are for a 95% confidence interval. nPRE and nPOST indicate the number of under-5s reflected in the PRE and POST results.

Comparison to external data sources

DHS & MICS

To sense-check our data, we compared our data points to well-recognised external surveys such as The Demographic Health Survey (DHS) and the Multiple Indicator Cluster Surveys (MICS). These surveys contain many key child health metrics including diarrhoea prevalence and ORSZ usage rates. Both of these surveys measure a 2-week prevalence for diarrhoea, with the timing of these surveys in Nigeria being DHS 2018 and MICS 2021. We reviewed the data specific to Kano state.

These data points give us some confidence in the accuracy of the data we collected, while acknowledging the limitations in generalisability of our data (refer to limitations section). We note that the urban slum ward had a disproportionately high prevalence at baseline at 30.1%; we consider the possibility that this may be due to a local outbreak (anecdotally provided).

ORS/Zinc/Both Usage rates, pre- and post-intervention

Next, we investigated our primary question, the impact of the intervention on ORSZ usage for diarrhoea in under-5s. The table below shows the proportion of children with diarrhoea in the 4 weeks preceding the survey who were treated with ORS, zinc, or both. Rate change measures the percentage point (as opposed to percentage) increase from the PRE-intervention group to the POST-intervention group.

In designing our questionnaire, we considered how different biases may play a role in ascertaining accurate information. Thus, “Unprompted’ (“How did you treat the diarrhoea?”) and “Prompted” (“Did you use ORS?”) question styles were used in this sequence to elicit treatment information. We hypothesised potential under-reporting bias from failure to recall or recognise ORS as a treatment for Unprompted, and over-reporting social desirability bias for Prompted.

Note: All margins of error shown are for a 95% confidence interval. These are unadjusted data. nPRE and nPOST indicate the number of under-5s’ last diarrhoea episode treatment reflected in the PRE and POST results.

Comparison to external data sources

We consider three comparable datasets: DHS 2018 (Nigeria), MICS 2021 (Nigeria), and Wagner et al 2019 study (Uganda). We use DHS and MICS to sense-check our baseline ORSZ usage rates, and the Uganda study to compare our effect sizes.

DHS & MICS

The Kano-specific numbers for ORS/Zinc usage rates are as follows. The comparable figures would be the PRE-intervention data for the prompted questions. Note that both these surveys measured in the 2-week time block in keeping with their diarrhoea prevalence data.

Our pilot data looks moderately comparable for the ORS usage rate but tends to be lower for the Zinc and ORSZ usage rates. Our pilot data seems consistent with our community members' focus groups, in which awareness and usage of zinc were markedly low. It is plausible that differences in the chance of sampling explain the slightly lower zinc usage rates seen in the urban, urban-slum, and peri-urban wards. The significant drop in this rate for the rural ward may be indicative of the impact of rurality on zinc usage.

Wagner et al 2019 (Uganda RCT 2019)

The Wagner et al. 2019 study in Uganda captured data for a time block of 4 weeks and is the main benchmark study for our analysis. This study compared four different ORS distribution methods: our comparisons of interest are the “free and convenient ” and control groups.

The free and convenient group closely reflects our pilot intervention group, with CHWs delivering ORSZ co-packs door-to-door for free prior to a child being ill. The control group differs slightly in that in the Wagner et al 2019 study, this was the ‘status quo’ where the CHWs sell ORS, zinc and other health products door-to-door once per month. In practice, only a minority of homes were visited each month. Similar services are not present in our pilot setting to the best of our knowledge, though there was some overlap with another free distribution program, ANRiN, which is currently winding down. We consider the implication of this in the next section.

The relevant Wagner results are as follows (copied from the abstract):

“During follow-up, … share of cases treated with ORS was… 77% (448/584) in the free and convenient group,... 56% (335/597) in the control group… After adjusting for potential confounders, instructing CHWs to provide free and convenient distribution increased ORS coverage by 19 percentage points relative to the control group (95% CI 13–26; P < 0.001)...”

In our pilot data, the most conservative increase in ORS usage rate was 43 pp (unadjusted), a remarkably high value compared to the Wagner et al. 2019 study of 19 pp (21 pp unadjusted). The Wagner study noted that “the intervention had a stronger effect among villages with particularly low ORS use at baseline”, and our pilot had a lower baseline (47% - 51%) compared to the Wagner study (56%). However, the magnitude of the difference is too large to be explained by this alone.

If our results approximate the realities in Nigeria, this is a significant opportunity with the potential for a very cost-effective intervention. However, we carefully consider the limitations of our pilot in a later section and how ‘discounts’ might be applied to the apparent effect size.

Overlap with another ORSZ distribution program

A nutrition project, ANRiN, also distributed ORSZ within Kano in a period overlapping with our pilot. Upon further analysis, 6.5% of the households in our pilot had interaction with both interventions. The urban ward only had one household reporting an ANRiN visit so we excluded this ward from the following analysis. We examined the difference in the proportion of ORS usage (for direct questions) grouped by whether the caregiver reported ANRiN interaction during the 6-week follow-up period.

We performed a chi-squared test on this contingency table which returned a p-value of 0.05075.  We do not want to assert any strong statistical conclusions from this given the numerous limitations described in a later section. However, this does nudge us to speculate that having additional co-packs and/or multiple exposures to staff promoting ORSZ could have a positive impact.

Reasons for non-usage of ORS & zinc

We also asked the caregivers about the reasons for not using ORSZ for their child’s diarrhoea in order to understand potential barriers. At Baseline the top cited reasons for this were as follows.

At Follow-Up, the high ORSZ usage rate and limited 6-week timeframe for eligible diarrhoea cases meant a much smaller sample, within which “Child refuses” was the main reason provided for ORS (22%) and zinc (24%).

These are interesting findings in light of the caregiver focus group noting ‘price’ as the main barrier. Though qualitative results need to be looked at more carefully (e.g. are there factors that people would be less inclined to mention directly), we can gather that all three elements of price, education, and pre-emptive ownership of ORSZ seem important.

Diversion

Since the intervention is based upon home administration of ORSZ when a child is ill, the caregiver has an opportunity to ‘divert’ ORSZ to users who are less vulnerable to diarrhoeal mortality, ie. older children or adults. Having learned of diversion as a potentially meaningful side-effect of positive awareness of ORS in the local context during focus groups, we sought to quantify it and explore mitigations.

We collected relevant data at two different points in the survey as outlined below. The following data are for “eligible” households in our intervention; ie. they have 1+ child under-5 and a primary caregiver home at the time of the visit. The data reflects a 6-week window from distribution to follow-up surveys.

% Eligible Households who had used ORS (any source) for a given age category

Note: This usage is for ORS from any source, with small chemist shops/stalls (“PPMVs”) being a common additional source. This data was collected at the beginning of the survey to determine the general pattern of ORS usage in a household regardless of its source.

This usage pattern (50-70% of eligible households using ORS for children under-5) is not concerning in and of itself, but raises the question of whether the freely distributed ORS was ‘diverted’. As a signal for this, we examine the following data.

% Clear Solutions ORS sachets used, % by age group 

Note: The ORS sachets here refer specifically to those provided as part of the co-packs from Clear Solutions. This data was collected at the end of the survey. Approximately 2000 ORS sachets were reported used during the 6-week period.

The proportion of ‘intended’ usage for children under 5 was 79-87%, whilst the ‘diversion’ rate was 13-21%.

An ORS sachet or co-pack used for an older child or adult is one that is not available to treat an ill under-5, our target population.  The primary basis for our impact evaluation is mortality aversion in the under-5 population. Thus, it is important to understand what proportion of our ORS is reaching that population, the distribution size and frequency necessary to provide good under-5 coverage, and the associated cost-effectiveness impact of the diversion.

We suspect that the 13-21% diversion rate observed here is likely to be an underestimation of the true rate because the importance of retaining and using the co-packs specifically for children under 5 was emphasised during distribution. Thus, we believe that caregivers were subject to social desirability bias (see “Limitations” section) which may yield an inflated proportion of the socially ‘correct’ answer, ie. under 5s, not other groups.

We think it is possible to influence this diversion rate with targeted behaviour change methods, perhaps modifying the CHW messaging or product packaging. This is something we will explore in our future programs.

Limitations

There are several limitations, beyond those already described in the body, to the pilot program that are important to keep in mind as we interpret these data.

Generalisability

Though we examined wards of differing ruralities, this represents a small geographic sample of the state of Kano. Thus, generalisability at a wider level (state, federal) could be limited. Furthermore, we only present summary-level data that are not adjusted for potential confounders.

Social Desirability Bias

A challenge in evaluating the ORSZ distribution intervention is that we are not able to directly observe treatment taking place, instead providing instruction and later capturing respondents’ treatment reports.

During follow-up surveys, when a caregiver is asked whether ORSZ was used for the child’s diarrhoea, the response can be subject to social desirability bias as caregivers may be tempted to provide answers they believe will be viewed favourably by others. Furthermore, caregivers may also believe that saying they have used our ORSZ will lead to more being given, or if not used, it may be taken away. We tried to mitigate this through a clear and repeated disclaimer that their response would not change how they are treated in the intervention.

Repeated sampling of the population

Survey respondents experienced between 1 and 4 interactions with staff from our pilot program (1 CHW and sampling in up to 3 survey rounds), which may have accentuated the social desirability bias. We have tried to mitigate this by not emphasising our origin and purpose of visits, but our data enumeration team found that many caregivers seemed to recognise the enumerators during follow-ups. We believe that this could bias the results more towards giving a “yes, I used ORS” response as caregivers may want to appear appreciative of the organisation that distributed and educated them on the benefits of this product.

This challenge can be meaningfully mitigated as our scale of operation becomes larger allowing us to sample a smaller number of participants from each community. Ideally, we would run a quasi-randomised design which can provide a more confident result than a pre-post comparison.

Statistical Assumptions Non-adherence

Our statistical tests are limited by the fact that the survey data does not adhere to the assumption of independent samples. We did not have distinct households in both pre and post-surveys, rather they overlap meaningfully given our high sampling rates. Furthermore, there are inherent correlations of population within the same communities. These factors could lead us to underestimate the true variance of our effect size estimates, resulting in unduly narrow confidence intervals.

Assumptions of pre/post comparison

The nature of the pre/post-test we conducted comes with many underlying assumptions. We elaborate on two assumptions that may be particularly applicable in our context.

The first assumption is that by attributing all the changes between our pre-group and post-group solely to our intervention, we are implying that no other factors played a role that may have influenced the ORSZ usage rate. The 6-week interval between the two surveys could be considered a relatively short time for major external changes that may impact our outcome. However, one clear example where this assumption does not hold is the overlap with the ANRiN program as mentioned earlier.

The second assumption is that the activity of the pre-survey would not have impacted the outcome of the post-survey. However, in our context, it seems plausible that more interaction with the program staff (including data enumerators) may act as a behaviour-changing activity. For example, by helping caregivers remember and believe the importance of ORSZ usage for children’s diarrhoea, we might assert additional influence upon the post-survey usage rates for those who were exposed to the pre-survey.

Factors supporting a large effect size

We outline above why our results may overestimate the intervention impact and it will be important to generously discount our effect size as a result. However, we also want to provide an alternative perspective that may support the genuineness of our large effect size.

It was often thought that one of the challenges of improving ORS uptake is people often do not see ORS as of ‘medicinal value’ and thus limiting its uptake. However, our formative research indicated that the communities we interviewed had positive attitudes towards ORS - so much so that adults would also use this product for various reasons beyond diarrhoea such as for strength, weakness, and dehydration. However, it was also noted that price was the main barrier to using ORS.

Whilst the above finding raised concerns regarding the potential diversion of our freely provided ORSZ co-packs, this also points to the fact that once the price barrier is removed, a very high usage rate of these products may not seem too far-fetched.

Learning and Conclusions

We are positively surprised by the apparent effect size of the intervention and the operational success that delivered it. We are very aware of the risks that come with surveys as a measure of reality, given recall bias, social desirability bias, and other errors that are impossible to fully remove. Nonetheless, we conclude that the intervention was successful in increasing ORSZ treatment of children under-5, with a “least favourable” result of 43pp increase (urban ward, high baseline ORS usage), and larger apparent effects in the other wards, for zinc, and for ORS and zinc in combination.

Diversion of co-packs to other age groups was reported and we consider this an area warranting further analysis and possibly dedicated research. At a reported 79+% of our ORS sachets used for under-5s, the loss would not impact cost-effectiveness unduly; but we must take care to establish a reliable signal for this, given the social factors at play in caregivers admitting usage outside of the target group. We explore possible options in the next section.

It is also likely that the initially high effect size we observe at the beginning of the program will diminish over time. This could be due to recall, as the novelty wears off, or just a natural phenomenon of behaviour change interventions. Thus, it will be important to understand the frequency and the number of co-packs that need to be distributed to sustain the program.

We have not yet performed a detailed cost-effectiveness analysis of the pilot: we will cover that in a later post. In the meantime, our general sense is that the pilot was lower cost per co-pack delivered than initially expected given our limited economies of scale. The total program cost (ie. excluding “HQ costs”, mostly co-founder salaries) of the pilot was approximately US$10,000. This includes a disproportionate spend (~50%) on surveying at much higher sampling rates than would be likely at scale.

Next steps

With such promising pilot results, we plan to take measured steps to scale-up, and build upon the learnings to date with operational variants aimed at improving the core intervention.

An important consideration for our 12-18 month plan is the GiveWell grant to CHAI for ORSZ distribution and RCT in Bauchi state. With coincidental proximity in timing and geography (neither we nor CHAI knew of each other’s plans until late 2023), we see a great opportunity to bring agility and innovation to optimise intervention cost-effectiveness (we discuss potential avenues below), in parallel to CHAI’s work on proving the foundation with the large-scale RCT.  We are optimistic for a fruitful collaboration that increases our collective impact significantly.

Operational variants and studies under consideration

As we plan for the remainder of 2024, we have ideas for operational variants or studies to improve cost-effectiveness or strengthen impact evaluation. These go beyond the extensive set of tactical improvements that we will take forward to the next intervention stage, and are presented here to give a flavour of the opportunities we are considering in parallel to the core.

1. Studying “worst case” diversion rates by providing co-packs without under-5 specific messaging, then surveying usage without a clear ‘correct answer’
2. Enabling caregivers to request ORSZ ‘top-ups’ via SMS, with turnarounds such that CHWs may verify the reported child’s illness.
3. Testing co-pack labelling variants (eg. colour, images) impact upon diversion rates.
4. Trialling community-centre distribution points for “top-ups” after the first round of door-to-door outreach. Eg. “Quarterly ORSZ Day”.

Short-term next steps

While we evaluate the ideas above, we are also planning for a refined execution of the core implementation at 2-3x scale in the next 3-4 months.

Our focus will be:

• Improving recruitment, training, and supervision approaches for the quality and scalability of CHWs.
• Improving diversion rate survey estimates with more coherent cross-checks and consistency checking between questions.
• Automating the processing of paper distribution tracking sheets (used by CHWs who do not have consistent access to or familiarity with smartphones) with photos and image processing.

Interested in learning more or supporting us?

Thank you for your interest!

Get in touch or follow our journey with our newsletter and social media.

We would love to hear from you with suggestions from related work, improvement ideas, or with introductions to folks who may be interested in collaborating or supporting us.

Volunteering & Paid Positions

We are not yet hiring for permanent positions, but will update our career notifications group when we do so. We especially welcome interest from, or introductions to, potential candidates for a role overseeing on-the-ground and partner operations at a state/regional level in northern Nigeria.

We have some capacity to work with volunteers who can support with outreach & social media, M&E methods and design, data analysis, and charity registration in UK or US. If you are interested in volunteering, please review the volunteer guide on our careers page.

We also have a need for specialist experience within Nigeria primary healthcare, community health intervention design, and build-out of SMS-based and voucher interventions.

Financial support

We project that we are funded until September 2024, but we have ambitious plans in mind for the final quarter, with a funding gap of approximately $100,000.

You can donate here any time, but for materials to support a larger funding decision, please do get in touch.

A huge thank you to everyone who has supported us to date, with partnership, funding, advice, and encouragement!

More detailed report, updates until June'24

A slightly more detailed version of this report is posted at clearsolutions.global/pilot. We expect to make additions to it until June'24 as time allows further analysis and reflection.