The following contains resources that I (Eleni) curated to help the AI Science team of AI Safety Camp 2023 prepare for the second half of the project, i.e., forecasting science capabilities. Suggestions for improvement of this guide are welcome.

Key points and readings

for forecasting in general:

What is a vignette?: https://www.lesswrong.com/posts/jusSrXEAsiqehBsmh/vignettes-workshop-ai-impacts
Start using Metaculus and Manifold (if you haven’t already)
Book review of Superforecasting
Actually possible: thoughts on Utopia

for forecasting AI in particular:

What is a hard take off?: “A hard takeoff (or an AI going "FOOM"^[2]) refers to AGI expansion in a matter of minutes, days, or months. It is a fast, abruptly, local increase in capability. This scenario is widely considered much more precarious, as this involves an AGI rapidly ascending in power without human control. This may result in unexpected or undesired behavior (i.e. Unfriendly AI). It is one of the main ideas supporting the Intelligence explosion hypothesis.
Read more:https://www.lesswrong.com/posts/tjH8XPxAnr6JRbh7k/hard-takeoff
What is a soft take off?: “A soft takeoff refers to an AGI that would self-improve over a period of years or decades. This could be due to either the learning algorithm being too demanding for the hardware or because the AI relies on experiencing feedback from the real-world that would have to be played out in real time.”
What is a sharp left turn?: the transition from a slower to a faster scaling regime as defined here (6:27 to 7:50).
Pivotal acts: acts that we, humans, take that make a big difference in terms of making x-risk less likely.
Pivotal acts from Math AIs: use AI to solve alignment at the formal/mathematical level.
Objection: AI alignment doesn’t seem like a purely mathematical problem - it requires knowledge about different aspects of reality that we haven’t been able to formalize (yet) e.g., how do agents (both human and artificial) represent values.
Follow-up objection: if we knew how to formalize these problems then we wouldn’t need much help from AI science models, i.e., if we knew what questions to ask and how to ask them.
Such questions would be: “can you solve this [input difficult for humans yet solvable mathematical puzzle e.g., a matri

...

Machine Learning For Scientific Discovery