Thanks for creating this post! Sharing some thoughts on the topic based on my experience creating and redteaming theories of change (ToCs) with various EA orgs (partly echoing your observations and partly adding new points; Two concrete project examples can be found here and here).

1. Neglectedness of ToC work (basically echoing your claim). Due to the non-pressing nature and required senior input, ToC/strategy work seems to get deprioritized very often, e.g., I have been deprioritizing updating our ToC for three months due to more pressing work. I think the optimal time spent thinking about your ToC/prioritization/strategy depends on the maturity of your project and is hard to get right, but based on my experience, most of us spend too little time on it (just like we tend to spend too little time exploring our career options as it is worth to invest 800 hours in our career planning when we can increase its impact by 1% in expectation). Assuming your org has ten staff, which work 200 days a year for 8 hours per day, you would want to invest 16k hours in figuring out how staff spends their time best if it is likely to result in a 1% impact increase
2. More than one ToC. I think most orgs should have a ToC on the org, team and individual level as well as for each main program/activity. It seems not optimal to work on something without even having thought through how this will change the world for at least 3min (and if there are alternatives, how you could achieve the same with less work)
3. Different levels of granularity. Depending on the purpose and the context of your ToC, you can have a three-row ToC (e.g., for small projects you are exploring), a flow-chart (e.g., to communicate the ToC of your org clearly, see examples in this post) and/or an exhaustive document showing lots of reasoning transparency, alternatives you considered etc. (e.g., to lay out the ToC of your research agenda)
4. Developing a ToC. One simplified approach to develop a ToC on an org level which worked well with some clients but always needs tailoring looks very roughly like this: (1) Map out all potential sources of value of today (and potentially in the future), (2) Prioritize them, (3) Create a flow chart for the most promising source of value (potentially include other sources of value or create several flow charts), (4) Think through the flow of impact/value end to end as a sanity check, (5) collect data (e.g., talk to experts, run small experiments) to reduce uncertainties, (6) re-iterate. See more here
5. Strategic implications and influencing decisions (previously mentioned but I think this is a point many people found useful and is not stressed enough in typical ToC literature). Your ToC should inform key decisions and ultimately how you allocate resources (your staff's time or money). I never experienced that we were certain about all the sources of value and the causal relationships between each step or did not have a hypothesis on how to have even more impact with an adapted or new program. So the ToC work always had at least some implications for allocating time. One great example is the Fish Welfare Initiative, which included resolving uncertainties around their ToC in their yearly priorities (See slides 13 and 24)
6. Areas for improvement. (1) Trying to map every casual pathway and not focusing on the most important ones, (2) Deferring too much to others on what they perceive as valuable, e.g., the target group, and not doing enough first principles/independent thinking and data collection, (3) Not considering counterfactuals at all and/or not considering that there are likely several counterfactual worlds, and (4) Not laying out key assumptions and uncertainties (previously mentioned in the post but seems valuable to highlight that this also reflects my experience) among other things

Note that I likely have a significant sample bias, as organizations are unlikely to reach out to me if they have enough time to think through their ToC. Additionally, please read this as "random thoughts which came to Jona's mind when reading the article" and not as "these are the X main things EA orgs get wrong based on a careful analysis". I expect to update my views as I learn more

Hmm. Obviously, career advice depends a lot on the individual and the specific context, all things equal, I tentatively agree that there is some value in having seen a large "functioning" org. I think many of these orgs have also dysfunctional aspects (e.g., I think most orgs are struggling with sexual harassment and concentration of formal and informal power) and that working at normal orgs has quite high opportunity costs. I also think that many of my former employers were net negative for some silly which I think are highly relevant, e.g., high-quality decision making 

Thanks for clarifying! I think Training for Good looked into "scalable management trainings", but had a hard time identifying a common theme, which they could work on (This is my understanding based on a few  informal chats. This might be outdated and I am sure they have a more nuanced take). Based on my experience, different managers seem to have quite different struggles which change over time and good coaching and peer support seemed to be the most time-effective interventions for the managers (This is based on me chatting occasionally to people and not based on proper research or deep thinking about the topic) 

What do you specifically mean by "maturing in management, generally"? I noticed that people  tend to have very different things in mind when they are talking about "Improving management in EA" so could be worth clarifying

Some shameless self-promotion as this might be relevant to some readers: I work at cFactual, a new EA strategy consultancy, where one of our three initial  services is to optimize ToC's and KPI's together with organizations. Illustrative project experience includes the evaluation of the ToC and design of a KPI for GovAI’s fellowship program, building a quantitative impact and cost-effectiveness model for a global health NGO,  internally benchmarking the impact potential of two competing programs of an EA meta organization with each other, doing coaching with a co-founder of a successful longtermist org around Fermi-estimates and prioritization of activities as well as redteaming the impact evaluation of a program of a large EA organization.

Thanks for highlighting this offer again and sharing your feelings, Catherine! 

I like how you highlight that the forum is just one element of EA. Personally, I also distinguish quite strongly between EA  as a question and set of evolving ideas and the EA community (which is obviously a part of EA). 

Historically, I found it super valuable to talk with you through various sensitive community-building considerations and benefited a lot from your experience managing countless tricky situations I wasn't even aware of. Thanks for doing that important and hard behind-the-scenes work! 

Thanks for sharing, Catherine! I apply many of your tips and agree that they are super useful. Additional questions I ask myself quite often:

1. What is the goal I want to achieve? This is the question which helps me to structure my thinking and approach the most 
2. Am I asking the right question? Next to regularly not thinking through all the options I have, I also realize often that I am not asking the question I really care about in the first place
   1. Can I make a prediction about my decision? This helps me a lot to keep track of my decisions e.g., at cFactual we have a "prediction of the week" to calibrate ourselves on outcomes we expect to see,  identify differences in reasoning about important topics among team members, ...
3. Do I weigh all arguments/considerations equally or do I believe one argument is 10x more relevant than others?

Some tools for group decision-making we use:

1. Our google doc company template has as default drop-downs to always indicate the status of the document, time spent, what the stage of the document is (Strawmen, key arguments or flashed out document) and a section for a few words on epistemic status ("braindump/ 5min of desk research/ I am an expert/..."). This helped us a lot to remain focused and have higher quality discussions with a time investment of 20sec
2. We try to quantify our preferences, e.g., Instead of saying: I am in favour of option A, we aim to write: I am 55% in favour of A. This helps us quite often to make a judgement call without forward and backward writing of comments

If there are larger decisions I want to think through more rigorously, I quite often use this mental structure as a starting point (and then adapt it): Recommendation/conclusion incl. my certainty in the conclusion, alternative options, my arguments for the recommendation, my arguments against, key uncertainties, key assumptions, downside risks and predictions

Probably stating the obvious for many here: I think the CFAR handbook also has great prompts for people who are interested in the topic

Quick update: we launched an EA-aligned strategy consultancy, partly motivated by this post and the feedback we received from our pilot projects: https://forum.effectivealtruism.org/posts/a65aZvDAcPTkkjWHT/introducing-cfactual-a-new-ea-aligned-consultancy-1

Thanks! Yes, feel free to DM me, if relevant.

Thanks for the question, Merlin. Please note that we have a small sample size and are still refining our models of what skillsets are most relevant for more EA-aligned consulting. 

Three things that have been useful: 1) Structuring problems, projects and meetings well, 2) Being able to switch quickly between different levels of abstraction quickly and constantly: Thinking carefully about a key assumption in an excel model in one moment and thinking about how the results change the big picture for a CEO in the next moment 3) More vaguely: Just having seen and worked with a lot of organization, projects and leaders, probably shaped our intuitions 

Three things that were less useful than I thought: 1) Executional speed - On the margin, we care much more about what we work on and getting it right compared to getting things done; 2) Qualitative data selection e.g., interviewing - during the pilot project we did a lot more independent thinking and then specific testing of key uncertainties with key stakeholders. In traditional consulting, we would have conducted interviews earlier to develop our hypotheses (which has the cost of becoming potentially an echo chamber); 3) Project planning - during all pilot projects we adapted our activities quite a lot based on our updated models, what would be most impactful