Cross-posted from Cold Takes
I have very detailed opinions on lots of topics. I sometimes get asked how I do this, which might just be people making fun of me, but I choose to interpret it as a real question, and I’m going to sketch an answer here.
You can think of this as a sort of sequel to Minimal-Trust Investigations. That piece talked about how investigating things in depth can be valuable; this piece will try to give a sense of how to get an in-depth investigation off the ground, going from “I’ve never heard of this topic before” to “Let me tell you all my thoughts on that.”
The rough basic idea is that I organize my learning around writing rather than reading. This doesn’t mean I don’t read - just that the reading is always in service of the writing.
Here’s an outline:
| Step 1 | Pick a topic |
| Step 2 | Read and/or discuss with others (a bit) |
| Step 3 | Explain and defend my current, incredibly premature hypothesis, in writing (or conversation) |
| Step 4 | Find and list weaknesses in my case |
| Step 5 | Pick a subquestion and do more reading/discussing |
| Step 6 | Revise my claim / switch sides |
| Step 7 | Repeat steps 3-6 a bunch |
| Step 8 | Get feedback on a draft from others, and use this to keep repeating steps 3-6 |
The “traditionally” hard parts of this process are steps 4 and 6: spotting weaknesses in arguments, trying to resist the temptation to “stick to my guns” when my original hypothesis isn’t looking so good, etc.
But step 3 is a different kind of challenge: trying to “always have a hypothesis” and re-articulating it whenever it changes. By doing this, I try to continually focus my reading on the goal of forming a bottom-line view, rather than just “gathering information.” I think this makes my investigations more focused and directed, and the results easier to retain. I consider this approach to be probably the single biggest difference-maker between "reading a ton about lots of things, but retaining little" and "efficiently developing a set of views on key topics and retaining the reasoning behind them."
Below I'll give more detail on each step, then some brief notes (to be expanded on later) on why this process is challenging.
My process for learning by writing
Step 1: pick a topic. First, I decide what I want to form an opinion about. My basic approach here is: “Find claims that are important if true, and might be true.”
This doesn’t take creativity. We live in an ocean of takes, pundits, advocates, etc. I usually cheat by paying special attention to claims by people who seem particularly smart, interesting, unconventionally minded (not repeating the same stuff I hear everywhere), and interested in the things I’m interested in (such as the long-run future of humanity).
But I also tend to be at least curious about any claim that is both “important if true” and “not obviously wrong according to some concrete reason I can voice,” even if it’s coming from a very random source (Youtube commenter, whatever).
For a concrete example throughout this piece, I’ll use this hypothesis, which I examined pretty recently: “Human history is a story of life getting gradually, consistently better.”
(Other, more complicated examples are the Collapsing Civilizational Competence Hypothesis; the Most Important Century hypothesis; and my attempt to summarize history in one table.)
Step 2: read and/or discuss (a bit). I usually start by trying to read the most prominent 1-3 pieces that (a) defend the claim or (b) attack the claim or (c) set out to comprehensively review the evidence on both sides. I try to understand the major reasons they’re giving for the side they come down on. I also chat about the topic with people who know more about it than I do, and who aren’t too high-stakes to chat with.
In the example I’m using, I read the relevant parts of Better Angels of our Nature and Enlightenment Now (focusing on claims about life getting better, and skipping discussion of “why”). I then looked for critiques of the books that specifically responded to the claims about life having gotten better (again putting aside the “why”). This led mostly to claims about the peacefulness of hunter-gatherers.
Step 3: explain and defend my current, incredibly premature hypothesis, in writing (or conversation). This is where my approach gets unusual - I form a hypothesis about whether the claim is true, LONG before I’m “qualified to have an opinion.” The process looks less like “Read and digest everything out there on the topic” and more like “Read the 1-3 most prominent pieces on each side, then go.”
I don’t have an easy time explaining “how” I generate a hypothesis while knowing so little - it feels like I just always have a “guess” at the answer to some topic, whether or not I even want to (though it often takes me a lot of effort to articulate the guess in words). The main thing I have to say about the “how” is that it just doesn’t matter: at this stage the hypothesis is more about setting the stage for more questions about investigation than about really trying to be right, so it seems sufficient to “just start rambling onto the page, and make any corrections/edits that my current state of knowledge already forces.”
For this example, I noted down something along the lines of: “Life has gotten better throughout history. The best data on this comes from the last few hundred years, because before that we just didn’t keep many records. Sometimes people try to claim that the longest-ago, murkiest times were better, such as hunter-gatherer times, but there’s no evidence for this - in fact, empirical evidence shows that hunter-gatherers were very violent - and we should assume that these early times fit on the same general trendline, which would mean they were quite bad. (Also, if you go even further back than hunter-gatherers, you get to apes, whose lives seem really horrible, so that seems to fit the trend as well.1)”
It took real effort to disentangle the thoughts in my head to the point where I could write that, but I tried to focus on keeping things simple and not trying to get it perfect.
At this stage, this is not a nuanced, caveated, detailed or well-researched take. Instead, my approach is more like: “Try to state what I think in a pretty strong, bold manner; defend it aggressively; list all of the best counterarguments, and shoot them down.” This generally fails almost immediately.
Step 4: find and list weaknesses in my case. My next step is to play devil’s advocate against myself, such as by:
- Looking for people arguing things that contradict my working hypothesis, and looking for their strongest points.
- Noting claims I’ve made with this property: “I haven’t really made an attempt to look comprehensively at the arguments on both sides of this, and if I did I might change my mind.”
(This summary obscures an ocean of variation. Having more existing knowledge about a general area, and more experience with investigations in general, can make someone much better at noticing things like this.)
In the example, my “devil’s advocate” points included:
- I’m getting all of my “life has gotten better” charts from books that are potentially biased. I should do something to see whether there are other charts, excluded from those books, that tell the opposite story.
- From my brief skim, the “hunter-gatherers were violent” claim looks right, and the critiques seem very hand-wavy and non-data-based. But I should probably read them more carefully and pull out their strongest arguments.
- Even if hunter-gatherers were violent, what about other aspects of their lives? Wikipedia seems to have a pretty rosy picture …
In theory, I could swap Step 4 (listing things I’d like to look into more) with Step 3 (writing what I think). That is, I could try to review both sides of every point comprehensively before forming my own view, which means a lot more reading before I start writing.
I think many people try to do this, but in my experience at least, it’s not the best way to go.
- Debates tend to be many-dimensional: for example, “Has life gotten better?” quickly breaks down into “Has quality-of-life metric X gotten better over period Y?” for a whole bunch of different X-Y pairs (plus other questions2).
- So if my goal were “Understand both sides of every possible sub-debate,” I could be reading forever - for example, I might get embroiled in the debates and nuances around each different claim made about life getting better over the last few hundred years.
- By writing early, I get a chance to make sure I’ve written down the version of the claim I care most about, and make sure that any further investigation is focused on the things that matter most for changing my mind on this claim.
- Once I wrote down “There are a huge number of charts showing that life has gotten better over the last few hundred years,” I could see that deep-diving any particular one of those charts wouldn’t be the best use of time - compared to addressing the very weakest points in the claim I had written, by going back further in time to hunter-gatherer periods, or looking for entirely different collections of charts.
Step 5: pick a subquestion and do more reading and/or discussing. One of the most important factors that determines whether these investigations go well (in the sense of teaching me a lot relatively quickly) is deciding which subquestions to “dig into” and which not to. As just noted, writing the hypothesis down early is key.
I try to stay very focused on doing the reading (and/or low-stakes discussion) most likely to change the big-picture claim I’m making. I rarely read a book or paper “once from start to finish”; instead I energetically skip around trying to find the parts most likely to give me a solid reason to change my mind, read them carefully and often multiple times, try to figure out what else I should be reading (whether this is “other parts of the same document” or “academic papers on topic X”) to contextualize them, etc.
Step 6: Revise my claim / switch sides. This is one of the trickiest parts - pausing Step 5 as soon as I have a modified (often still simplified, under-researched and wrong) hypothesis. It’s hard to notice when my hypothesis changes, and hard to stay open to radical changes of direction (and I make no claim that I’m as good at it as I could be).
I often try radically flipping around my hypothesis, even if I haven’t actually been convinced that it’s wrong - sometimes when I’m feeling iffy about arguing for one side, it’s productive to just go ahead and try arguing for the other side. I tend to get further by noticing how I feel about the "best arguments for both sides" than by trying from the start to be even-handed.
In the example, I pretty quickly decided to try flipping my view around completely, and noted something like: “A lot of people assume life has gotten better over time, but that’s just the last few hundred years. In fact, our best guess is that hunter-gatherers were getting some really important things right, such as gender relations and mental health, that we still haven’t caught up to after centuries of progress. Agriculture killed that, and we’ve been slowly climbing out of a hole ever since. There should be tons more research on what hunter-gatherer societies are/were like, and whether we can replicate their key properties at scale today - this is a lot more promising than just continuing to push forward science and technology and modernity.”
This completely contradicted my initial hypothesis. (I now think both are wrong.)
This sent me down a new line of research: constructing the best argument I could that life was better in hunter-gatherer times.
Step 7: repeat steps 3-6 a bunch. I tried to gather the best evidence for hunter-gatherer life being good, and for it being bad, and zeroed in on gender relations and violence as particularly interesting, confusing debates; on both of these, I changed my hypothesis/headline several times.
My hypotheses became increasingly complex and detailed, as you can see from the final products: Pre-agriculture gender relations seem bad (which argues that gender relations for hunter-gatherers were/are far from Wikipedia’s rosy picture, according to the best available evidence, though the evidence is far from conclusive, and it’s especially unclear how pre-agriculture gender relations compare to today’s) and Unraveling the evidence about violence among very early humans (which argues that hunter-gatherer violence was indeed high, but that - contra Better Angels - it probably got even worse after the development of agriculture, before declining at some pretty unknown point before today).
I went through several cycles of “I think I know what I really think and I’m ready to write,” followed by “No, having started writing, I’m unsatisfied with my answer on this point and think a bit more investigation could change it.” So I kept alternating between writing and reading, but was always reading with the aim of getting back to writing.
I finally produced some full, opinionated drafts that seemed to me to be about the best I could do without a ton more work.
After I had satisfied myself on these points, I popped back up from the “hunter-gatherer” question to the original question of whether life has gotten better over time. I followed a similar process for investigating other subquestions, like “Is the set of charts I’ve found representative for the last few hundred years?” and “What about the period in between hunter-gatherer times and the last few hundred years?”
Step 8: add feedback from others into the loop. It takes me a long time to get to the point where I can no longer easily tear apart my own hypothesis. Once I do, I start seeking feedback from others - first just people I know who are likely to be helpful and interested in the topic, then experts and the public. This works the same basic way as Steps 4-7, but with others doing a lot of the “noticing weaknesses” part (Step 4).
When I publish, I am thinking of it more like “I can’t easily find more problems with this, so it’s time to see whether others can” than like “This is great and definitely right.”
I hope I haven’t made this sound fun or easy
Some things about this process that are hard, taxing, exhausting and a bit of a mental health gauntlet:
- I constantly have a feeling (after reading) like I know what I think and how to say it, then I start writing and immediately notice that I don’t at all. I need to take a lot of breaks and try a lot of times to even “write what I currently think,” even when it’s pretty simple and early.
- Every subquestion is something I could spend a lifetime learning about, if I chose to. I need to constantly interrupt myself and ask, “Is this a key point? Is this worth learning more about?” or else I’ll never finish.
- There are infinite tough judgment calls about things like “whether to look into some important-seeming point, or just reframe my hypothesis such that I don’t need to.” Sometimes the latter is the answer (it feels like some debate is important, but if I really think about it, I realize the thing I most care about can be argued for without getting to the bottom of it); sometimes the former is (it feels like I can try to get around some debate, but actually, I can’t really come to a reasonable conclusion without an exhausting deep dive).
- At any given point, I know that if I were just better at things like “noticing which points are really crucial” and “reformulating my hypothesis so that it’s easier to defend while still important,” I could probably do something twice as good in half the time … and I often realize after a massive deep dive that most of the time I spent wasn’t necessary.
- Because of these points, I have very little ability to predict when a project will be done; I am never confident that I’m doing it as well as I could; and I’m constantly interrupting myself to reflect on these things rather than getting into a flow.
- Half the time, all of this work just ends up with me agreeing with conventional wisdom or “the experts” anyway … so I’ve just poured in work and gone through a million iterations of changing my mind, and any random person I talk to about it will just be like “So you decided X? Yeah X is just what I had already assumed.”
- The whole experience is a mix of writing, Googling, reading, skimming, and pressuring myself to be more efficient, which is very different and much more unpleasant compared to the experience of just reading. (Among other things, I can read in a nice location and be looking at a book or e-ink instead of a screen. Most of the work of an “investigation” is in front of a glowing screen and requires an Internet connection.)
I’ll write more about these challenges in a future post. I definitely recommend reading as a superior leisure activity, but for me at least, writing-centric work seems better for learning.
I’m really interested in comments from anyone who tries this sort of thing out and has things to share about how it goes!
This is great! I once worked at a think-tank-type place where we spent a lot of time refining our ideas through fierce criticism of each other's thinking and writing. Since leaving there, I've found that I miss that intellectual environment. There was a while when I was trying to approach political debates in this spirit but I found almost nobody else who wanted to do it. It's hard to recreate such an environment elsewhere because it's hard to find people who want to put time into it, and who also are prepared to let go of ideas in the face of evidence (or just better ideas). And doing it by yourself, as you point out here, is also really time-intensive and maybe less fun.
This was a great read, and in some ways quite relatable. I like writing about things that interest me, but often found it hard to have enough confidence in my ideas to write them down. This changed when I developed some new ideas around writing, which seem comparable to the ones you wrote in your article: writing is not so much the end-product of thinking, as much as it is a part of the process of thinking. I was inspired by Karl Popper, who said in one of his essays (I can’t recall wich one but it was featured in “All Life is Problem Solving”) that writing is the objectification or externalisation of thought, and by externalising our thoughts we put them “out there” for other people and ourselves to scrutinize. To me this is a very liberating idea: writing isn’t just for people who know what they think, but also (and maybe especially) for people who are trying to find out what they think.
Thanks for this Holden. I usually stop at step 3 and let other educate me on why I am wrong. That's because I am lazy.
I agree with Jeremy, it is very difficult to get the fierce criticism from smart people who are willing to dig in enough to be useful. We should create that space online.
Holden, you have made a good start with your pre-publication review team. I have enjoyed those (very early) exchanges and would enjoy thinking about how to expand them...
Thanks again for the work you do, Holden.
David B
Thanks for describing your process! 🙂
I’m curious to try to transfer this approach further to ’Learning by Modeling’. Not only as in mathematical modeling but whatever technology (writing is a technology in a way) best fits the project.
Excellent post. Thanks for writing and sharing. I skimmed it first. Then realized its significance and then re-read with paper-pen. Taking notes while reading, based on the question/hypothesis that I have in mind! I am sharing this with others and on social media. I will probably come again and again to this article in future!
Thanks Holden! I found this article useful because it shows a way we can organizing learning process in our information abundant world where I frequently find myself in the self deceptive state of having some knowledge just because I read something. But I think it is important understand that knowledge is not a feeling but some recall or correct opinion we can describe to ourselves or others. I found strange though that it seems to me you contrast such a learning to just reading without writing. I think that is already a common sense that passive reading is not learning.
Probably we can add here the details on how to make such a writing. A great guide for this is Zettelkasten system for note taking as used by Niklas Luhmann. It points out that we should have separate permanent and literature notes among other things. And it also makes the point that we should organize our reading around writing specifically for science papers or articles, use our own words, etc.
This is a great framework, and it clearly is incredibly productive based on your engaging output. But I sometimes feel as though you move too fast through what you reference in footnote 2 here – criteria. So for example your discussion recently about whether our times can still produce Beethoven-level artistic achievement sent me off wondering about what one would want to measure in answering that question. My own not-too-formed examples might look like this:
So, Pet Sounds? I think it's getting cred on 2 and 3, but on 1? Hey, I like The Beach Boys, but mainly it's nostalgia for how they captured a certain California dream at a certain cultural moment. Will our great grand children be moved? Hmm.
So these may not be great or well-articulated criteria, but my point is that zeroing in on the criteria deserves a fair amount of focus.
One more thought about the above: The possibilities for craft and innovation have multiplied a hundredfold across all the arts. People still write plays for the stage, compose music for orchestras, paint on canvas, and write novels for ink and paper. But the forms and platforms beyond those are endlessly rich and layered. The scale has changed by orders of magnitude. But on the level of human perspicacity and insight, the scale has not changed. If anyone writing today is more perceptive and articulate than Shakespeare, or Tolstoy, then they need a better agent.
So it may appear like I'm responded to the wrong post, but this criteria point seems critical to me. Not that you've ignored it, but that it is even more central to most of the questions you plumb than you have made it.
The hardest part of this, IMO, is that it seems like many, if not most, sources are biased. Even if not to an explicit ideology, then to the writer's personality. (And then there is the need for academics et al to constantly find something "new" to say.)
Thanks for this post.
Great post! I look forward to trying it. Is it safe to assume the same method applies to investigating and writing about the present and future?
I've found a benefit of learning by writing is figuring out what's actually not important at all. I wrote some COVID updates, and often things I was mad about- public communication, policies, etc were just not important when it came to writing down stuff in a way that helped other people navigate the risk environment.
Is this post generally encouraging participation in EA and engagement with EA ideas? (Learn about a topic, discuss it with others, draft a piece, gain feedback, re-draft the piece or discuss something else/related, ask people with similar expertise for draft comments, and continue to develop your perspectives and further writing interests?)
This can be quite fun, by the nature of solving some of the most important problems.
Thanks for this post! I especially enjoy your emphasis on creating a hypothesis as a "flag pole" that you can iteratively test your understanding against; I imagine this is a great way to embody Kant's advice of "Have the courage to use your own reason" as well as getting used to overcoming Confirmation Bias. Really cool suggestion that I'll be implementing, thank you!
It seems there is some interest in the public scrutiny of ideas when going through this type of research (from Jeremy, David B, and Laain) and I'd also enjoy having a space of idea scrutiny to get that feedback from outside. Would you all be open to an initial low investment testing of a space like this?
If we have common areas we'd like to investigate we can give each other our feedback (Step 4 and 8) to reduce the individual strain that Holden describes above. Maybe a lite test of this would be to have a small email thread bringing attention to topics that feedback is being asked on, and another area (slack/discord?) that allows the hashing out of said claims? I believe this could increase the group's familiarity with the work as well as reduce the strain of an individual. If this could be done at scale, I initially believe this could serve as a great method of community updating and involvement. Finally, this exploration is certainly acting in the "not a finished thought, but am discovering an answer" and am open to ideas.