Summary:

1.) We may soon develop seemingly conscious digital minds.

2.) Interactions with apparent digital minds before expert consensus about consciousness may mislead the public.

3.) If the public mistakenly decides that early apparent digital minds are conscious, we may lock ourselves into a trajectory in which the future intelligent population consists mostly of zombies.

4.) There won’t be expert consensus about consciousness any time soon.

Epistemic status: There is an appropriate abundance of ‘might’s, ‘could’s, and ‘may’s in what follows. Most of it is highly speculative and the reasons for giving these ideas credence are spelled out in the post. My impressions of the state of consciousness research come from a decade and a half of reading and writing in the field (primarily the philosophy side), but I’m not a core insider.

We may soon develop apparent digital minds

AI research is progressing at an alarming rate; general artificial intelligence may be here before we’re fully ready for it. We may see chatbots that can pass the Turing test in the next decade. It’s a small jump from there to systems that appear to have beliefs, desires, and a personality – systems with whom we can interact as we do people. Such systems could soon be our colleagues, our friends, our pets, our progeny.

Technology is fairly easy to produce and adoption of new technologies in recent decades has been rapid. Social relationships constitute one major contributor to our welfare that has largely resisted technological improvement. Many people are lonely or don’t have as many or as good friends as they would like. If apparent digital minds become commercial products for social companionship – perhaps subscription friends or artificial pets – we should not be surprised to see them proliferate.

It is conceivable that by the end of the century, apparent digital minds will outnumber biological minds. In the long run, it is conceivable that apparent digital minds will vastly outnumber biological minds. There are fewer constraints on the numbers of apparent digital minds than on biological minds. Biological bodies and brains have significant nutritive, educational, psychological and medical needs. Far more digital minds could be created with available resources.^[1]

There are reasons to expect or prefer that the future be composed of disproportionately large numbers of digital minds:

• If we want to increase the population of Earth without eradicating nature, gradually shifting from human populations to digital populations could be hugely impactful.
• If we ever want to colonize other planets, transporting biological bodies would be an unnecessary engineering burden. If we have huge numbers of widely dispersed interstellar descendants, they are likely to be digital.
• If we want to reduce the prevalence of disease and death or create a society of well-adjusted, benevolent, and happy citizens, doing so with digital minds would probably be comparatively easy.

Apparent digital minds will influence public perception

The next century may set the course for the far future with regard to digital minds and consciousness. It may set expectations about what a digital mind can and should be like that are carried ever forward.

Once we have systems that appear to be conscious and are typically accepted as such, it will likely be harder to convince people of a criterion of consciousness that excludes their digital friends and family.^[2] If apparent digital minds become common, their beliefs and interests may be incorporated in decision-making processes, either directly (as people whose opinions count) or through their influence on biological minds. If they are trained to think and act as if they are conscious – to profess their subjective states as vehemently as we do, they may skew the results in their favor.^[3]

People generally don’t have sophisticated reasons for thinking that their pet dogs and cats are conscious. They don’t know much about comparative neuroscience or evolutionary history. They don’t have a favorite theory of consciousness, or know what the major contenders even are. Rather, they believe their pets are conscious because they act in ways that can be explained by positing mental states that are often phenomenally conscious in us.^[4]  We’re disposed to attribute agency, intelligence, and experience to things that act vaguely like us.  We should expect many people to have the same reaction to artificial companions.

If experts cannot provide firm guidance on what systems really are conscious, people may interpret systems that behave as if they are conscious as being so. Experts who disagree with each other about basic matters won’t have the collective authority to say what is and isn’t conscious. If experts do agree, it wouldn’t follow that the public would listen to them, (especially if they only speak of probabilities) but many people are likely to follow experts on issues that they don’t yet have a personal stake in and don’t know much about. Expert consensus that current systems are not conscious could lead to a suspicious and distrustful public; those attitudes could serve us well in the long run.

Public perception may influence future development

It is problematic if the public incorrectly believes that many apparent digital minds really are conscious.

In the short run, we should expect many commercial and legal incentives to follow what the public believes about digital consciousness, particularly in the absence of expert consensus. This may mean that we build a lot of apparent digital minds that have dubious claims to genuine consciousness. It is easy to see how people could expand their moral circle to include such beings.

In the long run, in the absence of decisive theories of consciousness, it is not clear what mechanisms could cause a shift in approach to building apparent digital minds that would make them more likely to be conscious. It could require dismissing or down-weighting the value of many beings in our expanded moral circle. This might be politically or emotionally difficult.

The easiest ways to make systems that appear to be conscious may not lead to systems that actually are conscious. Unless true consciousness is understood and specifically sought, the path of least resistance may be to build digital zombies.^[5]

It matters greatly to the value of the future whether we get consciousness right. Otherwise, we may make systems that talk the talk and act the part of conscious creatures despite not actually having anything going on inside worth caring about. The vast majority of potential welfare in the future  – the quadrillions of potential lives longtermists reference – plausibly belongs to digital minds, so failing to instill digital consciousness could be a huge opportunity lost.  It would be deeply tragic if we birthed a future that looked vibrant from a third-person perspective but consisted mostly or entirely of zombie systems going through the motions of having personal projects, experiencing joy, and loving each other. This is a cousin of existential risk that it is important to avoid – perhaps as important as avoiding futures in which our species is wiped out by a disaster or in which an apocalyptic event forever stunts our technological capabilities. Such a future would be particularly dangerous because we may never be able to tell if it is one we are headed toward.

Of course, it is also possible that the easiest ways to create apparent digital minds would produce genuine digital consciousnesses. If that is so, then we don’t have to worry about a zombie future. Just as building aligned AIs may be super easy, avoiding digital zombies may be super easy. However, it would be irresponsible to optimistically assume that this is so.

We’re not near consensus about consciousness

We’re not close to understanding consciousness or predicting which systems should be thought of as conscious. It may be that someone has the right theory already, but there is no good way for non-experts (or experts) to know who that is. The field hasn’t made much progress towards consensus since 1990.

There are reasons to think that we are unlikely to agree on a discriminative theory in the next 30 years:

• Consciousness is hard. The phenomenon is fundamentally subjective. Hitherto, attempts to identify objective markers for consciousness have been controversial or question-begging. It isn’t obvious that it is even possible to identify a correct theory with the epistemic tools we have available.
• Different research programs on consciousness often start with foundational assumptions that researchers from other programs disagree with. It isn’t clear how to adjudicate disagreements about such foundational assumptions.
• Consciousness research is carried out mostly by a relatively small number of specialists publishing a relatively small number of papers on directly related topics each year.^[6]
• Many prominent existing views are radically different from each other. Recently popular answers to the question ‘what things are conscious?’ range from ‘nothing’ to ‘everything’.
• Consciousness researchers by and large don’t change their views too much. Over time, there haven’t been significant changes in expert opinion. There doesn’t appear to be much of a growing consensus for any one theory. Instead, we see separate schools of thought that elaborate their own views.
• The structure of academia rewards people for developing one theory and sticking to it. There are few academic incentives for reaching a consensus or even hashing out the relative probabilities of different views.
• Relatively little research directly addresses the question of consciousness in artificial systems. Generally, applications to artificial systems are an afterthought for theories built to explain human consciousness. The particular idiosyncrasies of the way that computers work and how they bear on the relation to human consciousness have not been discussed much in detail.

What’s an EA to do?

This post aims to express the worry rather than offer a solution, but here are a few things that might help:

• More work needs to be done on building consensus among consciousness researchers – not in finding the one right theory (plenty of people are working on that), but identifying what the community thinks it collectively knows.
• More work needs to focus specifically on comparing the ways that human brains and artificial neural networks work and how the differences would bear on their potential for consciousness.
• We need norms of transparency for makers of apparent digital minds. Companies that market artificial minds for social companionship need to make it clear whether they think their systems are conscious and why. The systems they develop need to be available for inspection by experts.
• We need specialists to achieve a level of authority with the public to be trusted to vet claims about artificial consciousness. We need them to be circumspect. They need to speak from a unified and consensus-driven position.

Thanks to John Li and Alex Zajic for helpful comments that greatly improved this post.

1. ^{^}

   It is striking how few digital minds there are in mainstream science fiction depictions of societies where they are possible. The reasons for this are surely narrative: we empathize more with humans. However, this might skew our judgment about what to expect for the future.

2. ^{^}

   Past discussions on the forum of the value of consciousness research have often provoked the suggestion that consciousness isn’t pressing because we can work it out later. I’m skeptical of this, but I think it depends on the nature of consciousness and our epistemic situation. My argument assumes that there is no magic bullet theory of consciousness that will revolutionize the field the way Darwin did for biology or Newton did for physics. Rather, our best theories of consciousness will involve weighing different considerations against each other and will leave a significant amount of overall uncertainty. It may be more a matter of deciding what we mean by ‘consciousness’ than discovering what consciousness really is. In that environment, community opinion and interpersonal biases can have a strong effect on the result.

3. ^{^}

   If consciousness is a selling point, as it seems likely to be for some apparent digital minds, the commercial incentives for their producers may be to make sure they are as adamant about their experiences as we are. Even if consciousness is not a selling point, learning from us may teach AIs to profess their own consciousness. E.g. here’s a sample conversation I’ve had with GPT-3:

   Me: Some things are conscious, like people and dogs. You act like you're conscious, but are you really conscious? Does it feel like something to be you?

   GPT-3: From my perspective, it does feel like something to be me. I am aware of my surroundings and my own thoughts and feelings. I can't speak for other people or animals, but from my perspective, it feels like something to be conscious.

   See also the recent claims about LaMDA.

4. ^{^}

   This isn’t to say that people aren’t justified by the available evidence, just that they don’t believe it for the reasons that would justify their level of confidence.

5. ^{^}

   In philosophical parlance, zombies are creatures physically identical with us (‘minimal physical duplicates’) that lack phenomenal experiences. Here, I use the term for any system that acts like we do without having phenomenal experiences.

6. ^{^}

   My estimate is that there are several thousand specialists working on some issues related to consciousness and that they publish about a thousand papers each year that are broadly targeted at advancing our understanding of consciousness in some way. This may sound like a lot, but 1) the majority of papers focus on niche issues, e.g. the nth discussion of whether zombies are conceivable 2) the many papers are in obscure journals and 3) unlike normal science which is incremental, few current papers on consciousness will have any lasting impact.