Residential pilot, 2021:

• “Building a Better Future”

• Students seemed brighter and more engaged with the content than I expected.
• Turns out that you can get really high quality applicants to programmes if you contact lots of different places, even with cold outreach methods, no demonstrable track record, and a very MVP website with stock images. The previous team contacted ~200 schools and received ~70 applications, of which they accepted 16.
• There was fuller attendance than I expected — all 16 applicants offered a place made it to the residential, whereas I had expected closer to ~50% drop out, as often happens with free events.
• (Also it seemed plausibly my comparative advantage given my experience in teaching and talent search / community building.)

Summer residential, 2022:

• “Building a Better Future, 2022”

• Try out slightly scaling what seemed like a successful model for talent search and community building.
• I took on Leaf, handed over from Alex Holness-Tofts; it was partly just about replicating the success of the pilot in the summer without doing anything too ambitious.

• Managed to identify a reasonably promising group of young people, and successfully encouraged some engagement with effective altruism and longtermism.
• The second BBF went roughly according to expectations, with various small updates in both positive and negative directions.
• There were a few unexpected forms of engagement in the immediate follow-up after the programme, but longer-term engagement was disappointing.
• It seemed like participants shifted their cause prioritisation views.
• The cold outreach methods that we used last time (emailing schools) didn’t scale as well as I had hoped; we only got 174 applicants.
• There was anecdotal evidence that some of the concerns and risks relating to outreach to high school audiences have indeed been borne out to some extent, e.g. some evidence of overwhelmingness.

Online early 2023:

• “How to choose a degree that helps you help others”

• “How to save 10 lives by age 20”

January to April 2023

(active March)

• I had resolved to test out a cheaper programme type than in-person residentials, to see if I could achieve a large fraction of the value for a lower cost.
• I wanted to test a uni/degree choice programme against one focused more directly on how to tackle the world’s most pressing problems.
• I also became excited about using them as talent search, funnelling, and filtering for the residentials (or other more resource-intensive follow-up programmes).

• Similar application numbers (138) and quality to our previous in-person programme.
• Degree programmes less appealing to applicants than expected.
• Degree programmes do affect degree choices!
• Potential for online programmes to sort by promise and engagement.

(I later concluded that some of the most promising participants in the Changemakers Fellowship had been identified by (and gained comparable value from) these programmes, and/or the Non-Trivial fellowship.)

Summer residential, 2023:

• “Changemakers Fellowship”

• I had resolved to test out ways to improve the effects per participant of the residential programme, without increasing costs by much.
• I ran quite a few variations to try to achieve this.
• I wanted to evaluate (cost-)effectiveness more rigorously, collaborating with Michael Noetel to run an experiment at 3-month follow-up, comparing between accepted and narrowly rejected applicants (and statistically controlling for differences between them).

• ~5x applicant numbers (755) and improved quality compared to the 2022 residential or the 2023 online programmes
• Some of the tweaks helped a bit, some made it worse.
• Broadly, the results of the more rigorous experimental follow-up study were disappointing. There was some evidence of positive changes, but less than you might hope/expect given the resource expenditure.
• Overall, usefulness per participant seemed lower than 2022 and my guess of cost-effectiveness fell (to ~one-third of the 2022 residential) despite the benefit of experience, track record for marketing, and more time spent on the programme.
• After the follow-up survey results I decided to indefinitely deprioritise residential programmes.

Subject-specific online programme pilot:

• “History to shape history (for the better)”

• I wanted to test the subject-specific format, which I was optimistic about, e.g. on application quality and quantity, e.g. for apparent effects.
• I had initially fundraised to run the subject-specific programmes as weekend-long in-person programmes, but pivoted to online due to the relative success of the early 2023 programmes and user feedback suggesting demand wouldn’t be sufficiently different to justify the higher costs.
• I wanted to test something relatively quick and cheap while I waited for the results of the Changemakers Fellowship follow-up survey.

• Disappointing application numbers (153); similar to the early 2023 online programmes, and ~0.2x the Changemakers Fellowship.
• Seemed fairly promising in terms of effects per participant (given the costs). E.g. frequency of change in university and career plans seemed similar to the Changemakers Fellowship; self-reported change in interest in EA and support for positively influencing the long-term future was higher.
• (I was feeling pessimistic about Leaf’s prospects at this point. I applied to several alternative jobs, but continued planning for Leaf next steps too.)

• I wanted to test an alternative option for supporting especially promising participants, either to create a new standalone programme with public applications or just as a follow-up support method for alumni of other programmes.
• I only ran this with 6 participants and 5 orgs, with no public application round; this was primarily through direct invite of Changemakers Fellows.
• (More info on the motivation here)

• Decent interest from potential partner organisations. I just dropped a Google Form on the Charity Entrepreneurship slack and this led to 6 sign ups. From directly contacting ~5 other orgs, I had 2 sign ups.
• Was harder than expected to get placements to actually start in earnest, which would make an organised cohort tough.
• As of now, 1 clear ‘success’ for the intern and their org; 1 seems weakly net negative for the org; 1 paused/stopped early; 1 never started; and 1 org (2 interns) only just starting but seems potentially promising.
• (More thorough data collection pending.)

Online early 2024:

• “The Mathematics of Morality”, aka MoM

• 2nd cohort of “History to shape history (for the better)”,  aka HSH

• “Dilemmas and Dangers in AI: How YOU can make a difference”, aka AI

• I felt surprised at the low application numbers for History to Shape History 2023 and suspected that a new programme might do better.
• I wanted to directly test subject-specific vs cause-specific online programmes, e.g. on application quality and quantity, e.g. for apparent effects.
• I wanted to test the potential for setting up an ecosystem of online programmes, where alumni sign up for follow-up online programmes.
• I wanted to test the scalability of the online programmes by reusing History to Shape History with extra marketing and alumni facilitators but little change to the content.
• I wanted to test smaller tweaks to the model, e.g. incorporating a weekly essay competition into HSH, e.g. to see if I could run AI with a much lower-effort curriculum.

• 10x the number of applications (1,563) as the 2023 online cohort and 2x the number of the Changemakers Fellowship, due to a mix of improved product, improved website, and an alumni TikTok which received 100k views. My optimism in the online fellowships as a scalable model was restored!
• MoM had 5x the applications that AI had, and 10x HSH. My optimism in the subject-specific model was restored!
• Promise for an ecosystem of Leaf online programmes: 68% of AI Fellows signed up for follow up fellowships (e.g. a Leaf AISF adaptation, switching to MoM), as did 32% of MoM and HSH fellows.
• Leaf alumni facilitators were great. I estimate that I can 1.5x the size of each fellowship each cohort without needing to hire facilitators externally.
• Likelihood to recommend was high (9.0 average for Fellows) across programmes, despite way less effort being put into the AI curriculum; this suggests the possibility to rapidly expand our array of online options.
• It may be difficult to run termly cohorts advertised to the same audience; there were fewer HSH applications than in late 2023, despite website improvements (but possibly due to applicants switching to MoM).
• 32% of HSH Fellows proposed independent follow-up projects; building towards impact-focused projects is feasible alongside subject-specific fellowships, with appropriate structuring.
• I raised £5.6k in donations from participants’ parents, and expect to be able to cover 10-40% of online programme financial costs this way going forwards.