Epistemic status: This is a pretty detailed hypothesis that I think overall doesn’t add up to more than 50% of my probability mass on explaining datapoints like FTX, Leverage Research, the LaSota crew etc., but is still my leading guess for what is going on. I might also be really confused about the whole topic.

Since the FTX explosion, I’ve been thinking a lot about what caused FTX and, relatedly, what caused other similarly crazy- or immoral-seeming groups of people in connection with the EA/Rationality/X-risk communities. 

I think  there is a common thread between a lot of the people behaving in crazy or reckless ways, that it can be explained, and that understanding what is going on there might be of enormous importance in modeling the future impact of the extended LW/EA social network.

The central thesis: "People want to fit in"

I think the vast majority of the variance in whether people turn crazy (and ironically also whether people end up aggressively “normal”) is dependent on their desire to fit into their social environment. The forces of conformity are enormous and strong, and most people are willing to quite drastically change how they relate to themselves, and what they are willing to do, based on relatively weak social forces, especially in the context of a bunch of social hyperstimulus (lovebombing is one central example of social hyperstimulus, but also twitter-mobs and social-justice cancelling behaviors seem similar to me in that they evoke extraordinarily strong reactions in people). 

My current model of this kind of motivation in people is quite path-dependent and myopic. Even if someone could leave a social context that seems kind of crazy or abusive to them and find a different social context that is better, with often only a few weeks of effort, they rarely do this (they won't necessarily find a great social context, since social relationships do take quite a while to form, but at least when I've observed abusive dynamics, it wouldn't take them very long to find one that is better than the bad situation in which they are currently in).  Instead people are very attached, much more than I think rational choice theory would generally predict, to the social context that they end up in, with people very rarely even considering the option of leaving and joining another one. 

This means that I currently think that the vast majority of people (around 90% of the population or so) are totally capable of being pressured into adopting extreme beliefs, being moved to extreme violence, or participating in highly immoral behavior, if you just put them into a social context where the incentives push in the right direction (see also Milgram and the effectiveness of military drafts). 

In this model, the primary reason for why people are not crazy is because social institutions and groups that drive people to extreme action tend to be short lived. The argument here is an argument from selection, not planning. Cults that drive people to extreme action die out quite quickly since they make enemies, or engage in various types of self-destructive behavior. Moderate religions that include some crazy stuff, but mostly cause people to care for themselves and not go crazy, survive through the ages and become the primary social context for a large fraction of the population. 

There is still a question of how you end up with groups of people who do take pretty crazy beliefs extremely seriously. I think there are a lot of different attractors that cause groups to end up with more of the crazy kind of social pressure. Sometimes people who are more straightforwardly crazy, who have really quite atypical brains, end up in positions of power and set a bunch of bad incentives. Sometimes it’s lead poisoning. Sometimes it’s sexual competition. But my current best guess for what explains the majority of the variance here is virtue-signaling races combined with evaporative cooling. 

Eliezer has already talked a bunch about this in his essays on cults, but here is my current short story for how groups of people end up having some really strong social forces towards crazy behavior. 

1. There is a relatively normal social group.
2. There is a demanding standard that the group is oriented around, which is external to any specific group member. This can be something like “devotion to god” or it can be something like the EA narrative of trying to help as many people as possible. 
3. When individuals signal that they are living their life according to the demanding standard, they get status and respect. The inclusion criterion in the group is whether someone is sufficiently living up to the demanding standard, according to vague social consensus.
4. At the beginning this looks pretty benign and like a bunch of people coming together to be good improv theater actors or something, or to have a local rationality meetup. 
5. But if group members are insecure enough, or if there is some limited pool of resources to divide up that each member really wants for themselves, then each member experiences a strong pressure to signal their devotion harder and harder, often burning substantial personal resources.
6. People who don’t want to live up to the demanding standard leave, which causes evaporative cooling and this raises the standards for the people who remain. Frequently this also causes the group to lose critical mass. 
7. The preceding steps cause a runaway signaling race in which people increasingly devote their resources to living up to the group's extreme standard, and profess more and more extreme beliefs in order to signal that they are living up to that extreme standard

I think the central driver in this story is the same central driver that causes most people to be boring, which is the desire to fit in. Same force, but if you set up the conditions a bit differently, and add a few additional things to the mix, you get pretty crazy results. 

Applying this model to EA and Rationality

I think the primary way the EA/Rationality community creates crazy stuff is by the mechanism above. I think a lot of this is just that we aren’t very conventional and so we tend to develop novel standards and social structures, and those aren’t selected for not-exploding, and so things we do explode more frequently. But I do also think we have a bunch of conditions that make the above dynamics more likely to happen, and also make the consequences of the above dynamics worse. 

But before I go into the details of the consequences, I want to talk a bit more about the evidence I have for this being a good model. 

1. Eliezer wrote about something quite close to this 10+ years ago and derived it from a bunch of observations of other cults, before really our community had shown much of any of these dynamics, so it wins some “non-hindsight bias” points.
2. I think this fits the LaSota crew situation in a lot of detail. A bunch of insecure people who really want a place to belong find the LaSota crew, which offers them a place to belong, but comes with (pretty crazy) high standards. People go crazy trying to demonstrate devotion to the crazy standard.
3. I also think this fits the FTX situation quite well. My current best model of what happened at an individual psychological level was many people being attracted to FTX/Alameda because of the potential resources, then many rounds of evaporative cooling as anyone who was not extremely hardcore according to the group standard was kicked out, with there being a constant sense of insecurity for everyone involved that came from the frequent purges of people who seemed to not be on board with the group standard.
4. This also fits my independent evidence from researching cults and other more extreme social groups, and what the dynamics there tend to be. One concrete prediction of this model is that the people who feel most insecure tend to be driven to the most extreme actions, which is borne out in a bunch of cult situations. 

Now, I think a bunch of EA and Rationality stuff tends to make the dynamics here worse: 

1. We tend to attract people who are unwelcome in other parts of the world. This includes a lot of autistic people, trans people, atheists from religious communities, etc.
2. The standards that we have in our groups, especially within EA, have signaling spirals that pass through a bunch of possibilities that sure seem really scary, like terrorism or fraud (unlike e.g. a group of monks, who might have signaling spirals that cause them to meditate all day, which can be individually destructive but does not have a ton of externalities). Indeed, many of our standards directly encourage *doing big things* and *thinking worldscale*.
3. We are generally quite isolationist, which means that there are fewer norms that we share with more long-lived groups which might act as antibodies for the most destructive kind of ideas (importantly, I think these memes are not optimized for not causing collateral damage in other ways; indeed, many stability-memes make many forms of innovation or growth or thinking a bunch harder, and I am very glad we don’t have them).
4. We attract a lot of people who are deeply ambitious (and also our standards encourage ambition), which means even periods of relative plenty can induce strong insecurities because people’s goals are unbounded, they are never satisfied, and marginal resources are always useful.

Now one might think that because we have a lot of smart people, we might be able to avoid the worst outcomes here, by just not enforcing extreme standards that seem pretty crazy. And indeed I think this does help! However, I also think it’s not enough because: 

Social miasma is much dumber than the average member of a group

I think a key point to pay attention to in what is going on in these kind of runaway signaling dynamics is: “how does a person know what the group standard is?”. 

And the short answer to that is “well, the group standard is what everyone else believes the group standard is”. And this is the exact context in which social miasma dynamics come into play. To any individual in a group, it can easily be the case that they think the group standard seems dumb, but in a situation of risk aversion, the important part is that you do things that look to everyone like the kind of thing that others would think is part of the standard. In practice this boils down to a very limited kind of reasoning where you do things that look vaguely associated with whatever you think the standard is, often without that standard being grounded in much of any robust internal logic. And things that are inconsistent with the actual standard upon substantial reflection do not actually get punished, as long as they look like the kind of behavior that looks like it was generated by someone trying to follow the standard.

(Duncan gives a bunch more gears and details on this in his “Common Knowledge and Social Miasma” post: https://medium.com/@ThingMaker/common-knowledge-and-miasma-20d0076f9c8e) 

How do people avoid turning crazy? 

Despite me thinking the dynamics above are real and common, there are definitely things that both individuals and groups can do to make this kind of craziness less likely, and less bad when it happens. 

First of all, there are some obvious things this theory predicts: 

1. Don’t put yourself into positions of insecurity. This is particularly hard if you do indeed have world-scale ambitions. Have warning flags against desperation, especially when that desperation is related to things that your in-group wants to signal. Also, be willing to meditate on not achieving your world-scale goals, because if you are too desperate to achieve them you will probably go insane (for this kind of reason, and also some others).
2. Avoid groups with strong evaporative cooling dynamics. As part of that, avoid very steep status gradients within (or on the boundary of) a group. Smooth social gradients are better than strict in-and-out dynamics.
3. Probably be grounded in more than one social group. Even being part of two different high-intensity groups seems like it should reduce the dynamics here a lot. 
4. To some degree, avoid attracting people who have few other options, since it makes the already high switching and exit costs even higher.
5. Confidentiality and obscurity feel like they worsen the relevant dynamics a lot, since they prevent other people from sanity-checking your takes (though this is also much more broadly applicable). For example, being involved in crimes makes it much harder to get outside feedback on your decisions, since telling people what decisions you are facing now exposes you to the risk of them outing you. Or working on dangerous technologies that you can't tell anyone about makes it harder to get feedback on whether you are making the right tradeoffs (since doing so would usually involve leaking some of the details behind the dangerous technology). 
6. Combat general social miasma dynamics (e.g. by running surveys or otherwise collapsing a bunch of the weird social uncertainty that makes things insane). Public conversations seem like they should help a bunch, though my sense is that if the conversation ends up being less about the object-level and more about persecuting people (or trying to police what people think) this can make things worse. 

There are a lot of other dynamics that I think are relevant here, and I think there are a lot more things one can do to fight against these dynamics, and there are also a ton of other factors that I haven’t talked about (willingness to do crazy mental experiments, contrarianism causing active distaste for certain forms of common sense, some people using a bunch of drugs, high price of Bay Area housing, messed up gender-ratio and some associated dynamics, and many more things). This is definitely not a comprehensive treatment, but it feels like currently one of the most important pieces for understanding what is going on when people in the extended EA/Rationality/X-Risk social network turn crazy in scary ways.