I (with lots of help from my colleague Marie Davidsen Buhl) made a database of resources relevant nanotechnology strategy research, with articles sorted by relevance for people new to the area. I hope it will be useful for people who want to look into doing research in this area.

Here are some forecasts for near-term progress / impacts of AI on research. They are the results of some small-ish number of hours of reading + thinking, and shouldn't be taken at all seriously. I'm sharing in case it's interesting for people and especially to get feedback on my bottom line probabilities and thought processes. I'm pretty sure there are some things I'm very wrong about in the below and I'd love for those to be corrected.

1. Deepmind will announce excellent performance from Alphafold2 (AF2) or some successor / relative for multi-domain proteins by end of 2023; or some other group will announced this using some AI scheme: 80% probability
2. Deepmind will announce excellent performance from AF2 or some successor / relative for protein complexes by end of 2023; or some other group will announced this using some AI scheme: 70% probability
3. Widespread adoption of a system like OpenAI Codex for data analysis will happen by end of 2023: 20% probability

I realise that "excellent performance" etc is vague, I choose to live with that rather than putting in the time to make everything precise (or not doing the exercise at all).

If you don't know what multi-domain proteins and protein complexes are, I found this Mohammed Al Quraishi blog very useful (maybe try ctrl-f for those terms), although maybe you need to start with some relevant background knowledge. I don't have a great sense for how big a deal this would be for various areas of biological science, but my impression is that they're both roughly the same order of magnitude of usefulness as getting excellent performance on single-domain proteins was (i.e. what AF2 has already achieved).

As for why:

80% chance that excellent AI performance on multi-domain proteins is announced by end of 2023

• Top reasons for event happening
  • Deepmind apparently wants to tackle protein complexes ("Based on this and DeepMind’s repeated assertions…") and multi-domain proteins seem like a stepping stone for that
  • Mohammed Al Quraishi already thought AF2 dealt with multi-domain proteins "just fine" in Dec 2020
  • It's an extremely similar problem to the one they already cracked (single domains) and it seems tractable
• Top reasons against
  • Maybe they won't announce it, because it's not newsworthy enough; or they'll bundle it with some bigger announcement with lots of cool results (resulting in delayed announcement)
• Other reasons against
  • In particular, the results from the next CASP competition will presumably be announced in December 2022; if they haven't cracked it by then, maybe we won't hear about it by end of 2023
    • They'd need to get there by April 2022  (I think that is the submission deadline for CASP)
  • Maybe it will turn out to be way less tractable than expected
  • Maybe Deepmind will have other, even more pressing priorities, or some key people will unexpectedly leave, or they'll lose funding, or something else unexpected happens
• Key uncertainties
  • Are rival protein folding schemes targeting this?

70% chance that excellent AI performance on protein complexes is announced by end of 2023

• Top reasons for event happening
  • Deepmind apparently wants to tackle protein complexes ("Based on this and DeepMind’s repeated assertions…")
  • Mohammed Al Quraishi already thought it dealt with multi-domain proteins "just fine" in Dec 2020, and these are a stepping stone to protein complexes
  • There is quite a lot of data (though 10x less than for single proteins)
    • Unsure whether transfer learning (I maybe using the wrong term) is relevant here?
• Top reasons against
  • Maybe even if it's done by say mid 2023 it won't be announced until after 2023 because of Deepmind's media strategy
    • In particular, targeting a CASP would seem to require the high performance to be achieved by mid 2022; maybe this is the most likely scenario in worlds where Deepmind announces this before end of 2023
    • (although if Deepmind doesn't get there by CASP15, it seems like another group might announce something in say 2023)
  • Protein complexes are (maybe?) qualitatively different to single proteins
• Other reasons against
  • Maybe the lack of data will be decisive
  • Maybe Deepmind's priorities will change, etc, as in noted above in the multi-domain case
• Key uncertainties
  • Are rival schemes targeting this?

20% chance of widespread adoption of a system like OpenAI Codex for data analysis by end of 2023

(NB this is just about data analysis / "data science" rather than about usage of Codex in general)

• My "best guess" scenario
  • OpenAI releases an API for data science that is cheap but not free. In its current iteration, the software is "handy" but not more than that. A later iteration, released in 2023, is significantly more powerful and useful. But by the end of 2023 it is still not yet "widely used".
• Some reasons against event happening
  • Maybe Codex is currently not that useful in practice for data analysis
  • I think OpenAI won't release it for free so it won't become part of the "standard toolkit" in the same way that e.g. RStudio has
  • Things like RStudio take a long time to diffuse / become adopted
    • E.g. my guess is ~5 years to get 25% uptake of for ipython notebook or rstudio by data scientists, or something like that
• Key uncertainties
  • How much are OpenAI going to push this on people?
    • How much are they pushing the data science aspect particularly?
  • Will this be ~free to use or will it be licensed?
  • How quickly will it improve? How often does OpenAI release improved versions of things?
  • How fast did ipython notebook/rstudio get adopted?
  • How much has the GPT-3 API been used so far?
• Some useful links
  • From the OpenAI website: "During the initial period, OpenAI Codex will be offered for free"
  • This guy at VentureBeat thinks that Microsoft will make the profits from Codex (but that doesn't seem to rule out it being widely used to assist with data analysis; just implies that the data analysis software will be owned by microsoft, I guess?)
  • The Verge says "although Codex is initially being released as free API, OpenAI will start charging for access at some point in the future."
  • The impression I get from Appendix H of OpenAI's paper on Codex is that they expect at least fairly wide adoption of Codex or something similar

Changing your working to fit the answer

I wrote this last Autumn as a private “blog post” shared only with a few colleagues. I’m posting it publicly now (after mild editing) because I have some vague idea that it can be good to make things like this public. It is quite rambling and doesn't really have a clear point (but I think it's at least an interesting topic).

Say you want to come up with a model for AI timelines, i.e. the probability of transformative AI being developed by year X for various values of X. You put in your assumptions (beliefs about the world), come up with a framework for combining them, and get an answer out. But then you’re not happy with the answer - your framework must have been flawed, or maybe on reflection one of your assumptions needs a bit of revision. So you fiddle with one or two things and get another answer - now it looks much better, close enough to your prior belief that it seems plausible, but not so close that it seems suspicious.

Is this kind of procedure valid? Here’s one case where the answer seems to be yes: if your conclusions are logically impossible, you know that either there’s a flaw in your framework or you need to revise your assumptions (or both). 

A closely related case is where the conclusion is logically possible, but extremely unlikely. It seems like there’s a lot of pressure to revise something then too.

But in the right context revising your model in this way can look pretty dodgy. It seems like you’re “doing things the wrong way round” - what was the point of building the model if you were going to fiddle with the assumptions until you got the answer you expected anyway?

I think this is connected to a lot of related issues / concepts:

• Model building
  • Option pricing models in finance: you start (both historically and conceptually) with the nice clean Black-Scholes model, which fails to explain actually observed option prices. Due to this, various assumptions are relaxed or modified, adding (arguably, somewhat ad hoc) complexity until, for the right set of parameters, the model gets all (sufficiently important) observed option prices right.
  • Regularisation / overfitting in ML: you might think of overfitting as “placing too much weight on getting the answer you expect”.
• Arguments
  • “One person's modus ponens is another’s modus tollens”: if we’re presented with a logical argument, usually the person presenting it wants us to accept the premises and agree that the argument is valid, in which case we must accept the conclusion. If we don’t like the conclusion, we often focus on showing that the argument is invalid. But if you think the conclusion is very unlikely, you also have the option of acknowledging the argument as valid, but rejecting one of the premises. There are lots of fun examples of this from science and philosophy on Gwern’s page on the subject.
  • “Begging the question”: a related accusation in philosophy that seems to mean roughly “your conclusion follows trivially from your premises but I reject one of your premises (and by the way it should have been obvious that I’d reject one of your premises so it was a waste of both my time and yours that you made this argument)”
  • Reductio ad absurdum: disprove something by using it as an assumption that leads to an implausible (or maybe logically impossible) conclusion
  • “Proving too much”: an accusation in philosophy that is supposed to count against the argument doing the “proving”.
  • (Not) updating your beliefs from an argument that appears convincing on the face of it: if the conclusions are implausible enough, you might not update your beliefs too much the first time you encounter the argument, even if it appears watertight.
• Research methods
  • Sanity checking your answer: check that the results of a complex calculation or experiment roughly match the result you get from a quick and crude approach.

Presumably, you could put this question of whether and how much to modify your model into some kind of formal Bayesian framework where on learning a new argument you update all your beliefs based on your prior beliefs in the premises, conclusion, and validity of the argument. I’m not sure whether there’s a literature on this, or whether e.g. highly skilled forecasters actually think like this.

In general though, it seems (to me) that there’s something important about “following where the assumptions / model takes you”. Maybe, given all the ways we fall short of being perfectly rational, we should (and I think that in fact we do) put more emphasis on this than a perfectly rational Bayesian agent would. Avoiding having a very strong prior on the conclusion seems helpful here.

Two papers I read on imprecise probabilities

I wrote this last Autumn as a private “blog post” shared only with a few colleagues. I’m posting it publicly now (after mild editing) because I have some vague idea that it can be good to make things like this public.

In this post I’m going to discuss two papers regarding imprecise probability that I read this week for a Decision Theory seminar. The first paper seeks to show that imprecise probabilities don’t adequately constrain the actions of a rational agent. The second paper seeks to refute that claim.

Just a note on how seriously to take what I’ve written here: I think I’ve got the gist of what’s in these papers, but I feel I could spend a lot more time making sure I’ve understood them and thinking about which arguments I find persuasive. It’s very possible I’ve misunderstood or misrepresented the points the papers were trying to make, and I can easily see myself changing my mind about things if I thought and read more.

Also, a note on terminology: it seems like “sharpness/unsharpness” and “precision/imprecision” are used interchangeably in these papers, as are “probability” and “credence”. There might well be subtle distinctions that I’m missing, but I’ll try to consistently use “imprecise probabilities” here.

Imprecise probabilities

I imagine there are (at least) several different ways of formulating imprecise probabilities. One way is the following: your belief state is represented by a set of probability functions, and your degree of belief in a particular proposition is represented by the set of values assigned to it by the set of probability functions. You also then have an imprecise expectation: each of your probability functions has an associated expected utility. Sometimes, all of your probability functions will agree on the action that has the highest expected value. In that case, you are rationally required to take that action. But if there’s no clear winner, that means there’s more than one permissible action you could take.

Subjective Probabilities should be Sharp

The first paper, Subjective Probabilities should be Sharp, was written in 2010 by Elga. The central claim is that there’s no plausible account of how imprecise probabilities constrain which choices are reasonable for a perfectly rational agent.

The argument centers around a particular betting scenario: someone tells you “I’m going to offer you bet A and then bet B, regarding a hypothesis H”:

Bet A: win $15 if H, else lose $10

Bet B: lose $10 if H, else win $15

You’re free to choose whether to take bet B independently of whether you choose bet A.

Depending on what you believe about H, it could well be that you prefer just one of the bets to both bets. But it seems like you really shouldn’t reject both bets. Taking both bets guarantees you’ll win exactly $5, which is strictly better than the $0 you’ll win if you reject both bets.

But under imprecise probabilities, it’s rationally permissible to have some range of probabilities for H, which implies that it’s permissible to reject both bet A and bet B. So imprecise probabilities permit something which seems like it ought to be impermissible.

Elga considers various rules that might be added to the initial imprecise probabilities-based decision theory, and argues that none of them are very appealing. I guess this isn’t as good as proving that there are no good rules or other modifications, but I found it fairly compelling on the face of it.

The rules that seemed most likely to work to me were Plan and Sequence. Both rules more or less entail that you should accept bet B if you already accepted bet A, in which case rejecting both bets is impermissible and it looks like the theory is saved. 

Elga tries to show that these don’t work by inviting us to imagine the case where a particular agent called Sally faces the decision problem. Sally has imprecise probabilities, maximises expected utility and has a utility function that is linear in dollars.

Elga argues that in this scenario it just doesn’t make sense for Sally to accept bet B only if she already accepted bet A - the decision to accept bet B shouldn’t depend on anything that came before. It might do if Sally had some risk averse decision theory, or had a utility function that was concave in dollars - but by assumption, she doesn’t. So Plan and Sequence, which had seemed like the best candidates for rescuing imprecise probabilities, aren’t plausible rules for a rational agent like Sally.

Should Subjective Probabilities be Sharp?

The 2014 paper by Bradley and Steele, Should Subjective Probabilities be Sharp? is, as the name suggests, a response to Elga’s paper. The core of their argument is that the assumptions for rationality implied by Elga’s argument are too strong and that it’s perfectly possible to have rational choice with imprecise probabilities provided that you don’t make these too-strong assumptions.

I’ll highlight two objections and give my view.

Objection 1:

• Bradley and Steele give the label Retrospective Rationality to the idea that an agent’s sequence of decisions should not be dominated by another sequence the agent could have made. They seem to reject Retrospective Rationality as a constraint on rational decision making because “[it] is useless to an agent who is wondering what to do… [the agent] should be concerned to make the best decision possible at [the time of the decision]”. 
• My view: I don’t find this a very compelling argument, at least in the current context - it seems to me that the agent should avoid foreseeably violating Retrospective Rationality, and in Elga’s betting scenario the irrationality of the “reject both bets” sequence of decisions seems perfectly foreseeable.

Objection 2:

• Their second objection is that Elga is wrong to think that your current decision about whether to accept bet B should be unaffected by whether you previously accepted or rejected bet A (they make a similar point regarding the decision to take bet A with vs without the knowledge that you’re about to be offered bet B). 
• My view: it’s true that, because in Elga’s betting scenario the outcomes of the bets are correlated, knowing whether or not you previously accepted bet A might well change your inclination to accept bet B, e.g. because of risk aversion or a non-linear utility function. But to me it seems right that for an agent whose decision theory doesn’t include these features, it would be irrational to change their inclination to accept bet B based on what came before - and Elga was considering such an agent. So I think I side with Elga here.

Summary and some thoughts

In summary, in Subjective Probabilities should be Sharp, Elga illustrates how imprecise probabilities appear to permit a risk-neutral agent with linear utility to make irrational choices. In addition, Elga argues that there aren’t any ways to rescue things while keeping imprecise probabilities. In Should Subjective Probabilities be Sharp?, Bradley and Steele argue that Elga makes some implausibly strong assumptions about what it takes to be rational. I didn't find these arguments very convincing, although I might well have just failed to appreciate the points they were trying to make.

I think it basically comes down to this: for an agent with decision theory features like Sally’s, i.e. no risk aversion and linear utility, the only way to avoid passing up opportunities like making a risk-free $5 by taking bet A and bet B is if you’re always willing to take one side of any particular bet. The problem with imprecise probabilities is that they permit you to refrain from taking either side, which implies that you’re permitted to decline the risk-free $5.

The fan of imprecise probabilities can wriggle out of this by saying that you should be allowed to do things like taking bet B only if you just took bet A - but I agree with Elga that this just doesn’t make sense for an agent like Sally. I think the reason this might look overly demanding on the face of it is that we’re not like Sally - we’re risk averse and have concave utility. But agents who are risk averse or have concave utility are allowed both to sometimes decline bets and to take risk-free sequences of bets, even according to Elga’s rationality requirements, so I don’t think this intuition pushes against Elga’s rationality requirements.

It feels kind of useful to have read these papers, because

• I’ve been kind of aware of imprecise probabilities and had a feeling I should think about them, and this has given me a bit of a feel for what they’re about.
• It makes further reading in this area easier.
• It’s good to get an idea of what sort of considerations people think about when deciding whether a decision theory is a good one. Similarly to when I dug more into moral philosophy, I now have more of a feeling along the lines of “there’s a lot of room for disagreement about what makes a good decision theory”.
• Relatedly, it’s good to get a bit of a feeling of “there’s nothing really revolutionary or groundbreaking here and I should to some extent feel free to do what I want”.

Some initial thoughts on "Are We Living At The Hinge Of History"?

In the below I give a very rough summary of Will MacAskill’s article Are We Living At The Hinge Of History? and give some very preliminary thoughts on the article and some of the questions it raises.

I definitely don’t think that what I’m writing here is particularly original or insightful: I’ve thought about this for no more than a few days, any points I make are probably repeating points other people have already made somewhere, and/or are misguided, etc. This seems like an incredibly deep topic which I feel like I’ve barely scratched the surface of. Also, this is not a focussed piece of writing trying to make a particular point, it’s just a collection of thoughts on a certain topic.

(If you want to just see what I think, skip down to "Some thoughts on the issues discussed in the article")

A summary of the article

(note that the article is an updated version of the original EA Forum post Are we living at the most influential time in history?)

Definition for the Hinge of History (HH)

The Hinge of History claim (HH): we are among the most influential people ever (past or future). Influentialness is, roughly, how much good a particular person at a particular time can do through direct expenditure of resources (rather than investment)

Two prominent longtermist EA views imply HH

Two worldviews prominent in longtermist EA imply that HH is true:

• Time of Perils view: we live at a time of unusually high extinction risk, and we can do an unusual amount to reduce this risk
• Value Lock-In view: we’ll soon invent a technology that allows present-day agents to assert their values indefinitely into the future (in the Bostrom-Yudkowsky version of this view, the technology is AI)

Arguments against HH

The base rates argument

Claim: our prior should be that we’re as likely as anyone else, past or present, to be the most influential person ever (Bostrom’s Self-Sampling Assumption (SSA)). Under this prior, it’s astronomically unlikely that any particular person is the most influential person ever.

Then the question is how much should we update from this prior

• The standard of evidence (Bayes factor) required to favour HH is incredibly high. E.g. we need a Bayes factor of ~10⁷ to move from a 1 in 100 million credence to a 1 in 10 credence. For comparison, a p=0.05 result from a randomised controlled trial gives a Bayes factor of 3 under certain reasonable assumptions.
• The arguments for Time of Perils or Value Lock-In might be somewhat convincing; but hard to see how they could be convincing enough
• E.g. our track record of understanding the importance of historical events is very poor
• When considering how much to update from the prior, we should be aware that there are biases that will tend to make us think HH is more likely than it really is

Counterargument 1: we only need to be at an enormously influential time, not the most influential, and the implications are ~the same either way

• Counter 1 to counterargument 1: the Bostrom-Yudkowsky view says we’re at the most influential time ever, so you should reject the Bostrom-Yudkowsky view if you’re abandoning the idea that we’re at the most influential time ever. So there is a material difference between “enormously influential time” and “most influential time”.
• Counter 2 to counterargument 1: if we’re not at the most influential time, presumably we should transfer our resources forward to the most influential time, so the difference between “enormously influential time” and “most influential time” is highly action-relevant.

Counterargument 2: the action-relevant thing is the influentialness of now compared to any time we can pass resources on to

• Again the Bostrom-Yudkowsky view is in conflict with this
• But MacAskill concedes that it does seem right that this is the action-relevant thing. So e.g. we could assume we can only transfer resources 1000 years into the future and define Restricted-HH: we are among the most influential people out of the people who will live over the next 1000 years

The inductive argument

• Claim: The influentialness of comparable people has been increasing over time, and we should expect this to continue, so the influentialness of future people who we can pass resources onto will be greater
• Evidence: if we consider the state of knowledge and ethics in 1600 vs today, or in 1920/1970 vs today, it seems clear that we have more knowledge and better ethics now than we did in 1600 or in 1920/1970
• And seems clear that there are huge gaps in our knowledge today (so doesn’t seem that we should expect this trend to break)

Arguments for HH

Single planet

Argument 1: we’re living on a single planet, implying greater influentialness

• Implies particular vulnerabilities e.g. asteroid strikes
• Implies individual people have an unusually large fraction of total resources
• Implies instant global communication

Counterargument:

• Asteroids are not a big risk
• For other prominent risks like AI or totalitarianism, being on multiple planets doesn’t seem to help
• We might well have quite a long future period on earth (1000s or 10,000s of years), which makes being on earth now less special
  • And in the early stages of space settlement the picture isn’t necessarily that relevantly different to the single planet one

Rapid growth

Argument 2: we’re now in a period of unusually fast economic and tech progress, implying greater influentialness. We can’t maintain the present-day growth rate indefinitely.

MacAskill seems sympathetic to the argument, but says it implies not that today is the most important time, but that the most important time is some time might be in the next few thousand years

• Also, maybe longtermist altruists are less influential during periods of fast economic growth because rapid change makes it harder to plan reliably
• And comparing economic power across long timescales is difficult

Other arguments

A few other arguments for HH are briefly touched on in a footnote: that existential risk / value lock-in lowers the number of future people in the reference class for the influentialness prior; that we might choose other priors that are more favourable for HH, and that earlier people can causally affect more future people

Some quick meta-level thoughts on the article

• I wish it had a detailed discussion about choosing a prior for influentialness, which I think is really important.
• There’s a comment that the article ignores the fact that the annual risk of extinction or lock-in in the future has implications for present-day influentialness because in Trammell’s model this is incorporated into the pure rate of time preference. I find that pretty weird. Trammell’s model is barely referenced elsewhere in the paper so I don’t really see why we should neglect to discuss something just because it happens to be interpreted in a certain way within his model. Maybe I missed the point here.
• I think it’s a shame that MacAskill doesn’t really give numbers in the article for his prior and posterior, either on HH or restricted-HH (this EA Forum comment thread by Greg Lewis is relevant).

Some thoughts on the issues discussed in the article

Two main points from the article

It kind of feels like there are two somewhat independent things that are most interesting from the article:

• 1. The claim: we should reject the Time of Perils view, and the Bostrom-Yudkowsky view, because in both cases the implication for our current influentialness is implausible
• 2. The question: what do high level / relatively abstract arguments tell us about whether we can do the most good by expending resources now or by passing resources on to future generations?

Avoiding rejecting the Time of Perils and Bostrom-Yudkowsky views

I think there are a few ways we can go to avoid rejecting the Time of Perils and Bostrom-Yudkowsky views

• We can find the evidence in favour of them strong enough to overwhelm the SSA prior through conventional Bayesian updating
• We can find the evidence in favour of them weaker than in the previous case, but still strong enough that we end up giving them significant credence in the face of the SSA prior, through some more forgiving method than Bayesian updating
• We can use a different prior, or claim that we should be uncertain between different priors
• Or we can just turn the argument (back?) around, and say that the SSA prior is implausible because it implies such a low probability for the Time of Perils and Bostrom-Yudkowsky views. Toby Ord seems to say something like this in the comments to the EA Forum post (see point 3).

A nearby alternative is to modify the Time of Perils and Bostrom-Yudkowsky views a bit so that they don’t imply we’re among the most influential people ever. E.g. for the Bostrom-Yudkowsky view we could make the value lock-in a bit “softer” by saying that for some reason, not necessarily known/stated, the lock-in would probably end after some moderate (on cosmological scales) length of time. I’d guess that many people might find a modified view more plausible even independently of the influentialness implications.

I’m not really sure what I think here, but I feel pretty sympathetic to the idea that we should be uncertain about the prior and that this maybe lends itself to having not too strong a prior against the Time of Perils and Bostrom-Yudkowsky views.

On the question of whether to expend resources now or later

The arguments MacAskill discusses suggest that the relevant time frame is the next few thousand years (because the next few thousand years seem (in expectation) especially high influentialness and because it might be effectively impossible to pass our resources further into the future).

It seems like the pivotal importance of priors on influentialness (or similar) then evaporates: it no longer seems that implausible on the SSA prior that now is a good time to expend resources rather than save. E.g. say there’ll be a 20 year period in the next 1000 years where we want to expend philanthropic resources rather than save them to pass on to future generations. Then a reasonable prior might be that we have a 20/1000 = 1 in 50 chance of being in that period. That’s a useful reference point and is enough to make us skeptical about arguments that we are in such a period, but it doesn’t seem overwhelming. In fact, we’d probably want to spend at least some resources now even purely based on this prior.

In particular, it seems like some kind of detailed analysis is needed, maybe along the lines of Trammell’s model or at least using that model as a starting point. I think many of the arguments in MacAskill’s article should be part of that detailed analysis, but, to stress the point, they don’t seem decisive to me.

This comment by Carl Shulman on the related EA Forum post and its replies has some stuff on this.

The importance of the idea of moral progress

In the article, the Inductive Argument is supported by the idea of moral progress: MacAskill cites the apparent progress in our moral values over the past 400 years as evidence for the idea that we should expect future generations to have better moral values than we do. Obviously, whether we should expect moral progress in the future is a really complex question, but I’m at least sympathetic to the idea that there isn’t really moral progress, just moral fashions (so societies closer in time to ours seem to have better moral values just because they tend to think more like us).

Of course, if we don’t expect moral progress, maybe it’s not so surprising that we have very high influentialness: if past and future actors don’t share our values, it seems very plausible on the face of it that we’re better off expending our resources now than passing them off to future generations in the hope they’ll carry out our wishes. So maybe MacAskill’s argument about influentialness should update us away from the idea of moral progress?

• This EA Forum comment by Larks and this one by Kbog are relevant.

But if we’re steadfast in our belief in moral progress, maybe it’s not so surprising that we have high influentialness because we find ourselves in a world where we are among the very few with a longtermist worldview, which won’t be the case in the future as longtermism becomes a more popular view. (I think Carl Shulman might say something like this in the comments to the original EA Forum post)

My overall take

• I think “how plausible is this stuff under an SSA prior” is a useful perspective
• Still, thinking about this hasn’t caused me to completely dismiss the Time of Perils View or the Bostrom-Yudkowsky view (I probably already had some kind of strong implausibility prior on those views).
• The arguments in the article are useful for thinking about how much (e.g.) the EA longtermist community should be spending rather than saving now, but a much more detailed analysis seems necessary to come to a firm view on this.

A quote to finish

I like the way the article ends, providing some motivation for the Inductive Argument in a way I find appealing on a gut level:

Just as our powers to grow crops, to transmit information, to discover the laws of nature, and to explore the cosmos have all increased over time, so will our power to make the world better — our influentialness. And given how much there is still to understand, we should believe, and hope, that our descendents look back at us as we look back at those in the medieval era, marvelling at how we could have got it all so wrong.

I recently spent some time trying to work out what I think about AI timelines. I definitely don’t have any particular insight here; I just thought it was a useful exercise for me to go through for various reasons (and I did find it very useful!).

As it came out, I "estimated" a ~5% chance of TAI by 2030 and a ~20% chance of TAI by 2050 (the probabilities for AGI are slightly higher). As you’d expect me to say, these numbers are highly non-robust.

When I showed them the below plots a couple of people commented that they were surprised that my AGI probabilities are higher than my TAI ones, and I now think I didn’t think about non-AGI routes to TAI enough when I did this. I’d now probably increase the TAI probabilities a bit and lower the AGI ones a bit compared to what I’m showing here (by “a bit” I mean ~maybe a few percentage points).

I generated these numbers by forming an inside view, an outside view, and making some heuristic adjustments. The inside and outside views are ~weighted averages of various forecasts. My timelines are especially sensitive to how I chose and weighted forecasts for my outside view.

Here are my timelines in graphical form:

And here they are again alongside some other timelines people have made public:

If you want more detail, there’s a lot more in this google doc. I’ll probably write another shortform post with some more thoughts / reflections on the process later.

Some initial thoughts on moral realism vs anti-realism

I wrote this last Summer as a private “blog post” just for me. I’m posting it publicly now (after mild editing) because I have some vague idea that it can be good to make things like this public. These thoughts come from my very naive point of view (as it was in the Summer of 2020; not to suggest my present day point of view is much less naive). In particular if you’ve already read lots of moral philosophy you probably won’t learn anything from reading this. Also, I hope my summaries of other people’s arguments aren’t too inaccurate.

Recently, I’ve been trying to think seriously about what it means to do good. A key part of Effective Altruism is asking ourselves how we can do the most good. Often, considering this question seems to be mostly an empirical task: how many lives will be saved through intervention A, and how many through intervention B? Aside from the empirical questions, though, there are also theoretical ones. One key consideration is what we mean by doing good.

Moral Philosophy

There is a branch of philosophy called moral philosophy which is (partly) concerned with answering this question. 

It’s important to me that I don’t get too drawn into the particular framings that have evolved within the academic discipline of moral philosophy, which are, presumably, partly due to cultural or historical forces, etc. This is because I really want to try to come up with my own view, and I think that (for me) the best process for this involves not taking other people’s views or existing works too seriously, especially while I try to think about these things seriously for the first time. 

Still, it seems useful to get familiar with the major insights and general way of thinking within moral philosophy, because

• I’ll surely learn a lot of useful stuff
• I’ll be able to communicate with other people who are familiar with moral philosophy (which probably includes most of the most interesting people to talk to on this topic).

Moral realism

I’ve read a couple of Stanford Encyclopedia of Philosophy articles, and a series of posts by Lukas Gloor arguing for moral anti-realism.

I found the Stanford Encyclopedia of Philosophy articles fairly tough going but still kind of useful. I thought the Gloor posts were great.

The Gloor posts have kind of convinced me to take the moral anti-realist side, which, roughly, denies the existence of definitive moral truths.

While I suppose I might consider my “inside view” to be moral anti-realist at the moment, I can easily see this changing in the future. For example, I imagine that if I read a well-argued case for moral realism, I might well change my mind. 

In fact, prior to reading Gloor’s posts, I probably considered myself to be a moral realist. I think I’d heard arguments, maybe from Will MacAskill, along the lines that i) if moral anti-realism is true, then nothing matters, whereas if realism is true, you should do what the true theory requires you to do, and ii) there’s some chance that realism is true, therefore iii) you should do what the true theory requires you to do.

Gloor discusses an argument like this in one of his posts. He calls belief in moral realism founded on this sort of argument “metaethical fanaticism” (if I’ve understood him correctly).

I’m not sure that I completely understood everything in Gloor’s posts. But the “fanaticism” label does feel appropriate to me. It feels like there’s a close analogy with the kinds of fanaticism that utilitarianism is susceptible to, for example. An example of that might be a Pascal’s wager type argument - if there’s a finite probability that I’ll get infinite utility derived from an eternal life in a Christian heaven, I should do what I can to maximise that probability. 

It feels like something has gone wrong here (although admittedly it’s not clear what), and this Pascal’s wager argument doesn’t feel at all like a strong argument for acting as if there’s a Christian heaven. Likewise, the “moral realist wager” doesn’t feel like a strong argument for acting as if moral realism is true, in my current view.

Moral anti-realism

Gloor also argues that we don’t lose anything worth having by being moral anti-realists, at least if you’re his brand of moral anti-realist. I think he calls the view he favours “pluralistic moral reductionism”. 

On his view, you take any moral view (or maybe combination of views) you like. These can (and maybe for some people, “should”) be grounded in our moral intuitions, and maybe use notions of simplicity of structure etc, just as a moral realist might ground their guess(?) at the true moral theory in similar principles. Your moral view is then your own “personal philosophy”, which you choose to live by.

One unfortunate consequence of this view is that you don’t really have any grounds to argue with someone else who happens to have a different view. Their view is only “wrong” in the sense that it doesn’t agree with yours; there’s no objective truth here.

From this perspective, it would arguably be nicer if everyone believed that there was a true moral view that we should strive to follow (even if we don’t know what it is). Especially if you also believe that we could make progress towards that true moral view.

I’m not sure how big this effect is, but it feels like more than nothing. So maybe I don’t quite agree that we don’t lose anything worth having by being moral anti-realists.

In any case, the fact that we might wish that moral realism is true doesn’t (presumably) have any bearing on whether or not it is true.

Reflections

I already mentioned that reading Gloor’s posts has caused me to favour moral anti-realism. Another effect, I think, is that I am more agnostic about the correct moral theory. Some form of utilitarianism, or at least consequentialism, seems far more plausible to me as the moral realist “one true theory” than a deontological theory or virtue ethics theory. Whereas if moral anti-realism is correct, I might be more open to non-consequentialist theories. (I’m not sure whether this new belief would stand up to a decent period of reflection, though - maybe I’d be just as much of a convinced moral anti-realist consequentialist after some reflection).

A moral philosophy free-for-all

I wrote this last Summer as a private “blog post” just for me. I’m posting it publicly now (after mild editing) because I have some vague idea that it can be good to make things like this public. These rambling thoughts come from my very naive point of view (as it was in the Summer of 2020; not to suggest my present day point of view is much less naive). In particular if you’ve already read lots of moral philosophy you probably won’t learn anything from reading this.

The free-for-all

Generally, reading various moral philosophy writings has probably made me (even) more comfortable trusting my own intuitions / reasoning regarding what “morality” is and what the “correct moral theory” is.

I think that, when you start engaging with moral philosophy, there’s a bit of a feeling that when you’re trying to reason about things like what’s right and wrong, which moral theory is superior to the other, etc, there are some concrete rules you need to follow, and (relatedly) certain words or phrases have a solid, technical definition that everyone with sufficient knowledge knows and agrees on. The “certain words or phrases” I have in mind here are things like “morally right”, “blameworthy”, “ought”, “value”, “acted wrongly”, etc.

To me right now, the situation seems a bit more like the following: moral philosophers (including knowledgeable amateurs, etc) have in mind definitions for certain words, but these definitions may be more or less precise, might change over time, and differ from person to person. And in making a “moral philosophy” argument (say, writing down an argument for a certain moral theory), the philosopher can use flexibility of interpretation as a tool to make their argument appear more forceful than it really is. Or, the philosopher’s argument might imply that certain things are self-evidently true, and the reader might be (maybe unconsciously) fooled into thinking that this is the case, when in fact it isn’t.

It seems to me now that genuinely self-evident truths are in very short supply in moral philosophy. And, now that I think this is the case, I feel like I have much more licence to make up my own mind about things. That feels quite liberating.

But it does also feel potentially dangerous. Of course, I don’t think it’s dangerous that *I* have freedom to decide what “doing good” means to me. But I might find it dangerous that others have that freedom. People can consider committing genocide to be “doing what is right” and it would be nice to have a stronger argument against this than “this conflicts with my personal definition of what good is”. And, of course, others might well think it’s dangerous that I have the freedom to decide what doing good means.

What does morality even mean?

Now that we’re in this free-for-all, even defining morality seems problematic.

I suppose I can make some observations about myself, like

• When I see injustice in the world, I feel a strong urge to do something about it
• When I see others suffering, I want to relieve that suffering
• I have a strong intuition that it’s conscious experience that ultimately matters - “what you don’t know can’t hurt you” is, I think, literally true
  • And some conscious experiences are clearly very bad (and some are clearly very good)
• And so on

I guess these things are all in the region of “wanting to improve the lives of others”. This sounds a lot like wanting to do what is morally good / morally praiseworthy, and seems at least closely related to morality.

In some ways, whether I label some of my goals and beliefs as being to do with “morality” doesn’t matter - either way, it seems clear that the academic field of moral philosophy is pretty relevant. And presumably when people talk about morality outside of an academic context, they’re at least sometimes talking about roughly the thing I’m thinking of.

Here are some thoughts after reading a book called "The Inner Game of Tennis" by Timothy Gallwey. I think it's quite a famous book and maybe a lot of people know it well already. I consider it to be mainly about how to prevent your system 2/conscious mind/analytical mind from interfering with the performance of your system 1/subconscious mind/intuitive mind. This is explained in the context of tennis, but it seems applicable to many other contexts, as the author himself argues. If that sounds interesting, I recommend checking the book out, it's short and quite readable.

My interest in the book comes mainly from thinking about the best way to go about doing research, at a day-to-day level. Although the arguments of the book seem most directly applicable to learning a physical skill/activity and (to some extent) to performing well at key moments, I still think there are lessons for mental activities performed routinely, i.e. for activities like research.

I think reading the book has generally pushed me a bit more in favour of "trusting my system 1/intuitive mind" while doing research, e.g. trusting that my brain is doing some important processing when I feel inclined to just stare into space and not make any apparent progress to whatever it is I'm trying to achieve at that moment. This feels pretty important.

I think Owen Cotton Barratt says some interesting things about trusting his intuition for prioritisation in this interview with Lynette Bye, which feels kind of related.

The book predates by many decades  Kahneman's Thinking Fast and Slow, which (I think) popularised the concept of system 1 mind and system 2 mind. The book instead refers to "self 1" and "self 2" which seem to have roughly similar meanings, although unfortunately reversed: Gallwey's self 1 and Kahneman's system 2 refer to the conscious/analytical mind, while Gallwey's self 2 and Kahneman's system 1 refer to the subconscious/intuitive mind.

Here are some disorganised notes on bits that seemed worth highlighting (page numbers refer to 2015 edition published by Pan Books):

• p13 mastering the mental side of tennis:
  • picture desired outcomes as clearly as possible
  • allow self 2 to perform and learn from successes and failures
  • learn to see non-judgementally: see what is happening rather than (just) seeing how well or badly it's happening
  • all subsidiary to the master skill: relaxed concentration
• p38 "Remember that you are not your tennis game. You are not your body. Trust the body to learn and to play, as you would trust another person to do a job, and in a short time it will perform beyond your expectations. Let the flower grow."
• p41 communicating with self 2
  • Gallwey exhorts the reader to trust their self 2 (system 1 / intuitive mind). But how can we be sure that self 2 will be optimising for the thing "we" (self 1) thinks is important? Gallwey gives 3 ways to convey to self 2 what the goal is, in the context of tennis:
    • Asking for results: visualise the exact path of the ball. Hold that image in your mind for several seconds
    • Asking for form: observe some particular aspect of your form (e.g. the flatness of your racket while it moves through the ball). Don't make an effort to make the change. Just visualise the change you want
    • Asking for qualities: imagine you are playing the role of a top tennis player on the court for a film
      • There are particular benefits of playing the role of someone very different to you
  • I'm not sure how to turn this into policies for doing research well. Things that seem interesting to explore:  visualising the output you want at the start of the day; reflecting each day on how what you did links to your ultimate goals; picturing yourself as playing the role of a researcher you admire.
• p80 on the "ego satisfaction" from a self-1-controlled success
  • Gallwey talks a lot about the ego satisfaction from self-1-controlled success. 
  • In the context of research, this doesn't seem to ring true for me from my experience (maybe it's obviously true for tennis or similar activities for people who have experience there, I don't know).
• p82 "Fighting the mind does not work. What works best is learning to focus it"
  • Gallwey talks about focussing on the seams of the ball and other techniques to focus the (self 1) mind on something kind of irrelevant while playing tennis so that the body and self 2 can perform without interference.
• p87 on what focus is: "Focus is not achieved by staring hard at something. It is not trying to force focus, nor does it mean thinking hard about something. Natural focus occurs when the mind is interested. When this occurs, the mind is drawn irresistibly toward the object (or subject) of interest. It is effortless and relaxed, not tense and overly controlled."
  • Re research, this seems like good advice for tackling a difficult problem or making progress on some task. One related thing is that I find it much easier to "effortlessly focus" on what I think is important if I'm free of distractions.
• p127 On managing stress. Pressures come at us from all corners: demands from partners, bosses, coaches, society, etc. These external demands can end up being internalised by self 1 and feeling as if they're things you really want, but this is an illusion.
  • (kind of reminds me of the message from another book I liked called Essentialism)