Following in the footsteps of LessWrong, we are running our first-ever EA Forum Review.
The purpose of the Review is to figure out which writings from the EA community have produced actual value, so that we can build on their ideas and share them with many more readers.
The Review uses the following process:
- Users cast preliminary votes to nominate posts they’ve found valuable for the final voting stage. (December 1 - December 14)
- Users review posts — explaining how the posts have been useful (or not), which of their ideas seem best (or worst), and so on. (December 15 - January 15)
- Users cast final votes on posts[1], based on their own experience and others’ reviews. The posts with the most votes are, hopefully, those that have provided the most collective value to the community. (January 16 - February 1) To see what this looks like in practice, check out LessWrong’s 2019 Review and its results.[2]
Why hold a review?
Effective altruism is focused on impact. But it’s hard to tell which intellectual work has actually made an impact.
The karma system shows us how many people liked a post right away. But that’s different from knowing what a post has achieved over the full span of its existence. Did it change the movement forever? Did people forget about it completely? Is it still useful?
The Review is our way of trying to answer questions like:
- “What are the most valuable things we’ve written? How might they be improved, or built upon?”
- “Which ideas are most important for someone to understand if they want to do as much good as possible? And where can they find the best writing on those ideas?”
- “What should people do if they want their own writing to be impactful? What can they learn by seeing trends and patterns in our best writing so far?”
Other benefits:
- This is a chance for people to share reflective commentary on important posts, with the benefit of hindsight. What still seems true, a year or five years later? Which ideas didn’t stand the test of time?
- It’s also a chance for authors to look back at their own posts and fix errors or make changes. The voting system lets them see which posts might be most impactful to update.
- To quote LessWrong: “There needs to be an incentive to clean up ideas that turned out to be important but badly presented.”
- Asking people to think about content they’ve valued might also lead them to crosspost it, making the Forum a more complete repository of EA’s intellectual contributions.
How does it work?
The Review will be visible to anyone who had a Forum account before December 1st, 2021.[3]
Our first review covers all posts published before 2021.[4] To quote LessWrong:
A year is probably enough time to no longer be swept away in the news or excitement of the day, but recent enough that we can still remember and write down how an idea or explanation has affected us.
(But don’t be shy about taking your best guess at how something much older affected you.)
Stage 1: Nominations / Preliminary Votes
From today through January 15th, eligible users will see this feature on eligible posts:
You can vote for as many posts as you like; a post will need at least one positive vote to be nominated to the second round. Negative votes don’t affect nomination status — only how posts are ranked.
The nomination scoring system
Here’s how we interpret each voting score[5]:
0 // Neutral: You have no strong feelings about the post.[6]
1 // Good: You found the post valuable. You’ve referred back to it before, or would recommend it to others.
4 // Important: Like “Good”, but more so. Perhaps the post includes one or more key insights that have changed the way you think, or led you to take useful actions.
9 // Crucial: The post is timeless — one of the most valuable on the Forum. You’d recommend it to almost everyone, and you think the community should be discussing and building on it for years to come.
-1 // Bad: The post is misleading, slightly harmful, or exceedingly unimportant. You wouldn’t personally recommend it to anyone.
-4 // Very bad: Like “Bad”, but more so. Perhaps the post includes one or more exceptionally bad ideas that have damaged the community, or led you to take actions you regret.
-9 // Awful: The post has had a worse effect on EA’s intellectual landscape than almost any other piece of writing.
How voting works
- You can vote for any eligible post, unless you were its author/co-author.
- You can cast an unlimited number of votes.
- However, the more votes you cast, and the higher your total “score” (where a “9” vote counts for 9x the score of a “1” vote), the less influential each of your votes will be. We normalize voting strength so that all users who are past a certain “score” threshold exert roughly the same amount of total influence.[7]
On the back end, we use a modified quadratic voting system, which allocates a fixed number of “points” across your votes based on how strong they are.
You can vote at the top of a post, or anywhere the post appears in a list (like the All Posts page).
You can also use the voting dashboard to see and vote on many older posts at once:
There’s also a new section on the front page:
The buttons on the bottom, from left to right, show you:
- Posts you’ve upvoted from before 2021 (a good way to find things to vote on for the review).
- A list of all eligible posts that haven’t yet been nominated.
- Posts that have already been nominated, so you can see what other people have been voting on.
Stage 2: Reviews
On December 14th, the list of nominated posts will close; no new posts can be nominated after that.
However, you can write reviews of posts. Here are some things you might discuss:
- How has this post been useful, to you or others you’ve known?
- How does it connect to the broader intellectual landscape of EA?
- Is this post epistemically sound? Does it say true things, and use language responsibly?
- Does the post still stand up to scrutiny today?
- How could the post be improved?
- How would you like to see someone build on this post with other work?
See these great examples from LessWrong’s 2018 review.
We hope that authors and other readers will find these comments valuable! To quote LessWrong:
Now is the time to give your opinions much more detail, distinguish between a post being an interesting hypothesis versus a robust argument, and generally help others understand what you think, so that we can discover exciting new disagreements and build much more robustly on past and future work.
Reviews can be voted on like other comments, but will have a special flair.
Only posts which have been reviewed with make it to the Final Voting phase.
What if some really good writing isn’t on the Forum?
- Crosspost it. If it was valuable to you, let’s make it easier to find!
- Add the review crosspost tag. This lets other people see all the new material that has gone up for review. If it was valuable to you, it may have been valuable to others!
- Contact us. Send an Intercom message or email forum@effectivealtruism.org, and we’ll backdate the post to its original publication date, making it eligible for nomination.
Stage 3: Final Votes
In the Review’s final phase, we hope that people will focus primarily on reading reviews, updating their posts, and finalizing their votes. (However, you’ll still be able to write new reviews.)
Voting will remain open through February 1st. You can only cast one vote for each post, but you’ll be able to change your votes until the review period ends.
Throughout the final voting period, the voting dashboard will continue to order posts by how many total “points” they’ve received.
What happens after the Review?
When the review ends, we’ll end up with a public list (on the voting dashboard) of which posts the Forum’s users found most valuable.
We’ll definitely create a sequence of these posts, and add it to our list of core readings. We may also follow LessWrong in creating a book, but that’s not guaranteed.
Prizes for great reviews and posts
What is guaranteed: Prizes!
We’re allocating up to $15,000 in total prize funding — $5,000 for the top posts, up to $5,000 for authors who update their posts during the Review, and up to $5,000[8] for the best reviews (as judged by a team of moderators and experienced authors).[9]
What should you do next?
Start finding posts to nominate and review!
Some ideas for how to start:
- Use our lists of posts you’ve upvoted before, and posts others have nominated.
- Look at posts that won the EA Forum Prize
- Find tags for topics you care about and look at the posts that use them.
- Check your history in your email, Messenger, or other apps to see which Forum posts you’ve shared (searching for “forum.effectivealtruism.org” should turn them up)
Questions and bug reports
Leave a comment if you have any questions about the review. (It’s a very detailed process, and I could easily have left out some helpful details in this post.)
We’re still working out a few bugs in the system, and changing some of our UI text to be more accurate. Leave a comment if you see anything that appears to need fixing!
Thank you, LessWrong!
The LessWrong team developed the concept for this review, did lots of coding to make it all work (including many changes this year), and wrote some excellent materials that made it much easier for me to write this post. We couldn’t have done this without them.
Or change their initial votes — you can only vote on each post once through the entire process. ↩︎
Note that we’ve combined the nomination and review periods, so our process looks a bit different than theirs. ↩︎
If you’ve been reading Forum content since before 2021, don’t have an account, and want to participate in the review, you can! Just make an account and email forum@effectivealtruism.org; we’ll make your account eligible. ↩︎
We say “2011-2020” in some places because this period covers all but five published Forum posts, the term “effective altruism” was coined in 2011, and “The Decade Review” has a nice ring to it. However, posts from before 2011 are also eligible. ↩︎
We wanted to include suggestions, but you are welcome to use your own definitions. ↩︎
We expect that you will feel this way about almost all of the Forum’s 4,000+ posts. ↩︎
We’ll share more specific figures on this later. ↩︎
We’re saying “up to” in case very few reviews are written or very few posts are updated. However, we think it’s likely that we’ll allocate the full $10,000 across these categories. ↩︎
The exact distribution of prize funding will depend on the specifics of how people allocate points, and how our judges feel about various reviews. ↩︎
I'm noticing that the EA Forum contains a lot of 'meta-level' content (i.e. discussions about the community, announcements, and similar), but a lot of object-level content (e.g. research reports, other forms of intellectual progress, or reports about successful EA projects) has been posted elsewhere.
I mainly want to upvote object-level content (and in fact have long felt pretty unhappy with the Forum's emphasis of meta-level content). But I don't really have time to crosspost more content myself. If someone wanted to do this, I'd be excited.
I'm willing to do a few more crossposts― are there pieces of object-level content that you'd really like to see crossposted?
Here's a somewhat random and non-exhaustive selection of (in my view) excellent content that's not on the Forum (disclosure: a lot is by CLR, the org I used to co-run):
Thanks for taking the time to put together this list, this is great! I found that a few of these were on the forum already:
I have crossposted the following, and may crosspost more if I feel like it (and will add them to this list if I do:
Also, to my pleasant shock, if you copy-paste from one website into the EA Forum WYSIWYG editor, it formats tables and images correctly? This makes cross-posting way easier than I'd realized!
Thanks for crossposting these. It seems that it's not possible to review or vote on some of those posts (specifically, these three posts). Is there an explanation for this? I also noticed I can't vote on this post by Carl Shulman, which I crossposted, though in that case I can write a review.
In general you're allowed to review, but not vote on your own posts.
Okay, that explains why I can't vote on the post by Carl I crossposted. But why can I (and everyone else, presumably) neither review nor vote on the three posts above?
They were not nominated during the nominations phase. I'll treat Tessa's posting as a nomination though, and nominate them manually. You should now be able to vote on and review them.
Very cool, thanks a lot! :)
FYI for anyone else who might crosspost Brian Tomasik posts: I learned thanks to a crosspost of The Importance of Wild-Animal Suffering that he doesn't like crossposting since it makes updating the content of posts more difficult. I have updated my crossposts from him to only include the summary paragraphs and a table of contents (with a caveat that the contents are as of the time of cross-posting).
Chloe Cockburn’s Summer 2020 Charities for Racial Justice: https://docs.google.com/document/d/1GGgEZ8ebFd6--C4wLeJV9XrX1OPPg40NL6F1QDo53Bs/edit
The Forum team has crossposted some of the best object-level content (e.g. most of Open Phil's blog, almost all FHI papers). However, there's far too much relevant object-level info scattered across far too many sources for us to capture everything. We appreciate all the users who take the time to share things they found valuable!
Just going over posts I've previously upvoted, I find it really remarkable how many impacted my thinking and actions, and how many I keep returning to. It's really inspiring :)
It seems to me that much of what richard_ngo highlighted in the EA Archives Reading List sequence should be available to vote on.
I could go through the list and crosspost what is neither already posted nor book-length/not-forum-friendly, but this sounds like a fair bit of work, so I am posting this comment to invite others to also makes some review crosspost posts based on that sequence.
What's the copyright situation here? If a crosspost is just a link to a post outside the EA forum, that's one thing; but if it involves copying the entire text to the forum, crossposting presumably requires the permission of the author.
Yes, you shouldn't crosspost the entire text of something without an author's permission.
But I think you can go beyond a link and share a brief excerpt of basically anything — paper abstracts are public, book reviews often quote extended sections of a book, etc.
In cases where the Forum's team has crossposted full articles, we check in with the author before doing so (e.g. for the Replacing Guilt sequence).
Realistically, though, we've had hundreds of things crossposted here by other users and have only had complaints in two cases I can remember — in both cases, when a private Google Doc was shared without the consent of all the people who'd been working on it. So we don't proactively remove types of content that seem very safe (e.g. posts from bloggers in the EA community who share all their content for free on their blogs), though we would remove something like a link to a pirated book if we became aware of it.
I am extremely happy that this is happening. And an annual review would be an excellent successor to the Forum Prize.
Edit: looks like this happened, thanks!
Request: let authors review their own posts.
This request is motivated by the fact that I can't review my linkposts, but more generally, I think it could conceivably be useful for authors to share their current thoughts on their old posts in a way that is particularly visible to those participating in the Review — that is, I think it could be useful for authors to review their own posts.
(Or maybe authors already can, but those posts just have to be nominated first? In that case, can someone nominate my linkposts for me?)
If anyone is wondering what is Quadratic Voting:
There is a good explanation from Lesswrong of how Quadratic Voting is used in the Forum Review.
Succinct explanation:
In Quadratic Voting, you can vote more for a single choice. Since you can vote more, you can express your degree of agreement, which probably captures preferences and the Truth better.
The quadratic part of Quadratic Voting makes sure that your additional votes for the same item costs increasingly more. This basically ensures voting multiple times won't break down (tempers extremes cases).
Wikipedia has more information.
Background:
What's sort of neat is that interest in Quadratic voting is probably the work of one person, Glen Weyl.
Loosely speaking, Weyl seems to be adjacent to EA, in the same way Vitalik Buterin is, who is his collaborator (Management Science is a top journal that is hard to publish in and it is impressive the outsider Buterin has a pub there). Both Buterin and Weyl are mentioned on EA Forum and Lesswrong a bit.
I'm uncertain, but I think the highly technical aesthetic of Weyl and Buterin's work and the related small social movement diverges from mainstream politics and even many mainstream economists. Tractability of their policies seems low according to this post and karma of related posts is low.
However, mechanism design is a neat field. It's neat to see a modest implementation of these ideas that is well designed and implemented by internal experts of a community.
Really cool, Weyl might like to know!
The voting mechanism here seems to sort of be the other way around. A "9" vote counts as 9 "1" votes, where as in QV it'd cost 9 to get 3 times "1".
Yes, you are referring to this:
Yes, like you said, these numbers [1,4,9] are how many times you vote for each entry.
The quadratic voting being implemented here isn't really connected to the values of these options. Instead, a calculation happens after you select your votes, e.g. when you choose the "vote 9 times" option, it will cost you more than 900% than "voting 1 time".
So I think that the set of voting options you see above is arbitrary. The choices could be [1,2,3] or [1,10,100] instead of [ 1,4,9].
The actual values just happen to squares. I guess that is sort of a "mental collision" with quadratic voting.
Maybe the designers (at LessWrong) choose 1,4,9, because these values are the best, or because it is sort of a cute callback to quadratic voting.
I noticed that when I go to the voting dashboard, I can't see this post listed there – is that a bug?
Thanks for this report — we're looking into it.
I can no longer see any of the posts I've voted on on the dashboard.
Can you share a screenshot of what you see instead?
We have extended the end of the review phase to the 15th, to give a full month for reviews.
It's nice that I can sort not-yet-nominated posts by karma (and it would be even nicer if the default setting for not-yet-nominated posts was sorted by karma, and filtered all-time rather than by year). Similarly valuable would be sorting nominated posts by karma when voting, unless it's intentional for some reason that nominated posts are in no order (or are they?).
I can be relatively OCD about voting, but it's important to me to either:
Based on the current UI, it looks like I can't not vote for a post after I've voted for it. And with quadratic voting, if my votes are normalized in certain ways (e.g., such that their mean is zero), then there is a big difference between not voting and voting zero.
(I assume votes won't be problematically normalized; e.g., if someone only gives upvotes, it should boost all of those posts relative to the posts they don't vote on. But wanted to make sure.)
I'm checking in with the LW team about these questions — I'll update this answer and send you a DM to notify you when I have an update.
Last call for reviews! You can still submit reviews after the end of this phase, but if you want your reviews to count for inclusion in the next phase, you'll need to submit the reviews before midnight UTC the night of the 15th. That's midnight GMT, or 7pm EST.
The timeline says there's preliminary voting until December 14th (already passed) and final voting starts January 12th. What about the time in between (e.g., now)?
If I vote now, will my votes be counted for the final vote (assuming I don't revise them later), or should I wait until January 12th?
Today's the last day of the Nominations Phase!
Suggestion/request: all past Forum Prize winners should be automatically nominated.
We considered doing this, but eventually decided it felt wrong for the spirit of the event.
We've had 573 posts nominated so far, so people are clearly finding lots of content they liked. If none of the Forum's users think to nominate a given Prize winner, that seems to indicate it wasn't especially impactful after the fact, and thus shouldn't be automatically nominated.
I have two questions:
Thank you!
Thank you for your crossposting contributions Tessa!
Am I right in thinking that posts with the 'Community' tag are not eligible?
No, they should be eligible.
I've updated the post to clarify that people can't vote on their own posts — maybe that's what you were seeing? Otherwise, what's the post where you don't see voting available?
Ahh, actually the issue is I forgot the 'not during 2021' restriction; as such everything seems to be working fine!
I hope that in the review phase, we'll be able to see preliminary votes? For example, if there's an old moderately popular post that I believe is quite wrong, I would probably want to write a high-effort negative review on it if and only if it was significantly upvoted in the nomination stage. Presumably that's the point of having the nomination phase before the review phase? I'd appreciate if this was clarified, though.
Edit: just read the complementary post on LessWrong, and it looks like the answer is yes.
Update: it's the review stage, and I can't see how much posts were upvoted. (By the way, how are they sorted?) I would like to be able to so that I can better prioritize review-writing.
For example, one post that I strong-downvoted was nominated, and I'd like to review it iff it received more than a couple upvotes (or if it gets positive reviews).
(LW Developer here: there's a code update ready-to-ship that updates the /reviewVoting page to show the outcome. It's been a bit delayed in merging roughly because JP and I are in different timezones)