*Cross-posted from my blog*

Even if we don't takeoff very quickly we still have to deal with potential Superintelligence.

There are lots of important uncertainties around intelligence that impact our response,  these are called by the community crucial considerations. The interlocking (non-exhaustive) list of crucial considerations list I tend to think about are things like:

• Will man-made Intelligence be sufficiently like current machine learning system so that you can expect safety solutions for ML to be at all transferable to it?
• Will man-made intelligence be neat or messy (and how does that impact the takeoff speed)
• Will a developing intelligence (whether a pure AI or human/machine hybrid) be able to get a decisive strategic advantage? Can they do so without war?
• Can we make a good world with intelligence augmentation or is co-ordination too hard?
• Should we expect a singleton to be stable? 

These cannot be known ahead of time, until we have developed intelligence or got pretty far down that road. We can estimate the current probabilities with our current knowledge but new information is coming in all the time.

Answering all these questions is an exercise in forecasting, making educated guesses. The better we can reduce the uncertainty around these questions, the better we can allocate resources to making sure the future of intelligence is beneficial to all. 

It is worthwhile to look at how forecasting is best done (with the caveat that even the best forecasting isn't very good at looking more than 5 years out). The state of the art, as lots of people are probably familiar, is Superforecasting, a methodology to follow developed by the Good Judgement Project. It is worth having a brief overview of what the Good Judgement Project was, so we can get an idea of why it might be worthwhile adopting it's lessons and why they might not apply.

Good Judgement Project

What they did 

From wikipedia:

The study employed several thousand people as volunteer forecasters.^[13] Using personality-trait tests, training methods and strategies the researchers at GJP were able to select forecasting participants with less cognitive bias than the average person; as the forecasting contest continued the researchers were able to further down select these individuals in groups of so-called superforecasters. The last season of the GJP enlisted a total of 260 superforecasters.

What they found to work

What they found to work was groups of people with diverse viewpoints of the world but a certain way of thinking. That way of thinking (take from the end of the book) was:

• Looking at both statistical probabilities of events and the specifics of one event to create an updated scenario.
• Looking at a problem from multiple different view points and synthesising them?
• Updating well after getting new information
• Breaking the problem down into sub-problems (fermi) that can be estimated numerically and combined
• Striving to distinguish as many degrees of doubt as possible - be as precise in your estimates as you can
• Learning to forecast better by doing it
• Work well in teams - understanding others view points

How relevant is it?

It may not be relevant at all. It was focused on questions that were explicitly just about tractable. At a sweet spot between too hard and too easy. We are dealing with questions that are too hard and too far out. But we don't really have anything better. Even prediction markets can not help us as we can't resolve these questions before hand and judge betters accuracy on problems. 

What lessons should we learn?

So how well is the community doing at following the lessons of Superforecasting? I shall focus on the Open Philanthropy Project and 80,000 hours, because they seem indicative of the Effective Altruism's response in general.

In part, I can't tell, I am led to believe that discussion has moved off-line and into private forums. If outsiders can't tell the thought processes behind the decisions then they cannot suggests improvements or alternative views to synthesize. What can we tell from their actions and public statements? I shall use three sources.

• Potential risks advanced artificial intelligence: The philanthropic opportunity - as it states the reasons for donating
• List of the donations - to get an idea of the relative weightings it gives to different problem areas
• Advice to people who want to work on AI safety technology and policy

So what can we see from these documents

• A focus on the concrete problems and a lack of investment in the novel
• A focus on AI rather than IA
• A narrow range of relevant subjects suggested for study, especially on the technical side
• A lack of a break down of the open questions

Let us take each in turn.

Focus on concrete problems

It seems that work is heavily weighted to concrete technical problems now. 38 million of 43 million dollars has been donated to teams looking at current AI techniques.  The rest has been donated to policy (FHI) or reasoning theory (MIRI). There is no acknowledgement anywhere that current AI techniques might not lead directly to future AI. Also this quote goes against them integrating multiple view points. From the strategy document. 

We believe that AI and machine learning researchers are the people best positioned to make many assessments that will be important to us, such as which technical problems seem tractable and high-potential and which researchers have impressive accomplishments.

This strategy is not getting a diverse view point to help forecaster what is likely to be useful. So this is quite worrying for how good we think their predictions will be about their subjects (considering that experts did worse predicting their events in their own fields compared to fields outside their own in the Good Judgement Project). It might be that the Open Philanthropy Project thinks that only AI projects based on current ML are currently technically tractable. But if that was the case you would expect more weight on AI policy/research as they might be able to gather more perspectives on what intelligence is (and how it might be made), I would also expect more calls for novel technical approaches.

Counterpoint

Reading the grant for OpenAI:

In fact, much of our other work in this cause aims primarily to help build a general field and culture that can react to a wide variety of potential future situations, and prioritizes this goal above supporting any particular line of research.

So the grant to OpenAI seems to be mainly about culture building. So the question is if current ML work does not lead to AI directly, e.g. if there is something like my resource allocation paradigm or something not yet thought of needed for AGI as the base-layer, will OpenAI  and the team their adopt it and lead the field?  Does OpenAI provide the necessary openness to multiple view points and exploration of other possibilities missing from the grants themselves? From what I can tell as an outsider it is only interested in deep learning. I would be pleased to learn otherwise though!

A focus on AI with no IA

While Superintelligence is bearish on Intelligence augmentation as a solution to the AI problem, that doesn't mean that it should not be considered at all. We may gain more knowledge that makes it more promising. But if you only search in one field, you will never get knowledge about other fields so you can update your probabilities. One of the super forecasters explicitly chose to get his news from randomised sources, in order to not be too biased. Funding lots of AI people and then asking them how their projects are going is the equivalent of getting information from one news source. There are IA projects like neuralink and kernel , so it is not as if there are no IA projects. I don't think either of them are looking at the software required on the computer to make a useful brain prosthesis (or whether we can make one that is not physically connected) at the moment, so this seems like a neglected question.

A narrow range of relevant subjects suggested for study

Currently 80,000 hours does not suggest learning any psychology or neuroscience for technical subjects. You can quite happily do machine learning without these things, but if there is a probability that current machine learning is not sufficient then the technical AI safety community needs something to steer itself with ourside of knowledge of machine learning. Kaj sotala has suggested psychology is neglected so there has been some discussion. But the reply was that ML is more useful for technical work right now for AI safety, which is true. This might be a tragedy of the commons problem of career choice, which I think was high-lighted in William Macaskill's speech, which I've not watched yet. There is a need for a diverse people set of people working on AI safety in the long term, but the best way of getting ahead is to optimize for the short term, which could lead to everyone knowing the same things and group think. On this point, to reduce this potential problem I think we should suggest technical people take a minor interest as well (be it cognitive science/psychology/sociology/neuroscience/philosophy/computer hardware) and try and integrate their knowledge of that with their normal ML or IA work. Encourage this by trying to make sure teams have people with diverse backgrounds and sub-interests so that they can make better predictions about where their work will go.

A lack of breakdown of the open questions and a current lack of updating

The only mention of probability in the strategy document is

I believe there’s a nontrivial probability that transformative AI will be developed within the next 20 years, with enormous global consequences.

And it is not quantified or broken down. We don't have fermi style breakdown (as recommended by super forecasting) of what he thinks the questions we need answering to increase or decrease our confidence.  He mentions at least two other points where there might be more details forthcoming.

I’ve long worried that it’s simply too difficult to make meaningful statements (even probabilistic ones) about the future course of technology and its implications. However, I’ve gradually changed my view on this topic, partly due to reading I’ve done on personal time. It will be challenging to assemble and present the key data points, but I hope to do so at some point this year.

Looking through the blogs I can't see the breakdown I also cannot see the update mentioned below

In about a year, we’ll formally review our progress and reconsider how senior staff time is allocated.

These two things that they had suggested that would happen would be very interesting to see and might enable us to see more of the internal workings and give better criticisms.

Suggestions

In order to have more accurate forecasts, I would like to see Open Philanthropy Project consult not just AI and ML viewpoints when trying to forecast what will be important work or the important organisations. 

They should look at allocating resources, finding advisers and perhaps building networks within the psychology and neuroscience communities and also possibly the nascent IA community to get their viewpoints on what intelligence is. This will enable them to update on what they think is the more probable approach to solving the super intelligence problem.

80,000 hours should encourage people studying technical AI subjects to have a broad background in other intelligence related fields as well as ML so that they can have more information on how the field should shift, if it is going to.

I'd also encourage Open Philanthropy Project to send out the data points and updates that they said they were going to do.

Whatever we end up forecasting about what we should do about the future of intelligence, I want to try and navigate a way through that improves human's freedom and autonomy, whilst not neglecting safety and existential risk reduction.