New & upvoted

Customize feedCustomize feed
· 1y ago · 2m read
· 3mo ago · 1m read

Posts tagged community

Quick takes

(COI note: I work at OpenAI. These are my personal views, though.) My quick take on the "AI pause debate", framed in terms of two scenarios for how the AI safety community might evolve over the coming years: 1. AI safety becomes the single community that's the most knowledgeable about cutting-edge ML systems. The smartest up-and-coming ML researchers find themselves constantly coming to AI safety spaces, because that's the place to go if you want to nerd out about the models. It feels like the early days of hacker culture. There's a constant flow of ideas and brainstorming in those spaces; the core alignment ideas are standard background knowledge for everyone there. There are hackathons where people build fun demos, and people figuring out ways of using AI to augment their research. Constant interactions with the models allows people to gain really good hands-on intuitions about how they work, which they leverage into doing great research that helps us actually understand them better. When the public ends up demanding regulation, there's a large pool of competent people who are broadly reasonable about the risks, and can slot into the relevant institutions and make them work well. 2. AI safety becomes much more similar to the environmentalist movement. It has broader reach, but alienates a lot of the most competent people in the relevant fields. ML researchers who find themselves in AI safety spaces are told they're "worse than Hitler" (which happened to a friend of mine, actually). People get deontological about AI progress; some hesitate to pay for ChatGPT because it feels like they're contributing to the problem (another true story); others overemphasize the risks of existing models in order to whip up popular support. People are sucked into psychological doom spirals similar to how many environmentalists think about climate change: if you're not depressed then you obviously don't take it seriously enough. Just like environmentalists often block some of the most valuable work on fixing climate change (e.g. nuclear energy, geoengineering, land use reform), safety advocates block some of the most valuable work on alignment (e.g. scalable oversight, interpretability, adversarial training) due to acceleration or misuse concerns. Of course, nobody will say they want to dramatically slow down alignment research, but there will be such high barriers to researchers getting and studying the relevant models that it has similar effects. The regulations that end up being implemented are messy and full of holes, because the movement is more focused on making a big statement than figuring out the details. Obviously I've exaggerated and caricatured these scenarios, but I think there's an important point here. One really good thing about the AI safety movement, until recently, is that the focus on the problem of technical alignment has nudged it away from the second scenario (although it wasn't particularly close to the first scenario either, because the "nerding out" was typically more about decision theory or agent foundations than ML itself). That's changed a bit lately, in part because a bunch of people seem to think that making technical progress on alignment is hopeless. I think this is just not an epistemically reasonable position to take: history is full of cases where people dramatically underestimated the growth of scientific knowledge, and its ability to solve big problems. Either way, I do think public advocacy for strong governance measures can be valuable, but I also think that "pause AI" advocacy runs the risk of pushing us towards scenario 2. Even if you think that's a cost worth paying, I'd urge you to think about ways to get the benefits of the advocacy while reducing that cost and keeping the door open for scenario 1.
(Clarification about my views in the context of the AI pause debate) I'm finding it hard to communicate my views on AI risk. I feel like some people are responding to the general vibe they think I'm giving off rather than the actual content. Other times, it seems like people will focus on a narrow snippet of my comments/post and respond to it without recognizing the context. For example, one person interpreted me as saying that I'm against literally any AI safety regulation. I'm not. For a full disclosure, my views on AI risk can be loosely summarized as follows: * I think AI will probably be very beneficial for humanity. * Nonetheless, I think that there are credible, foreseeable risks from AI that could do vast harm, and we should invest heavily to ensure these outcomes don't happen. * I also don't think technology is uniformly harmless. Plenty of technologies have caused net harm. Factory farming is a giant net harm that might have even made our entire industrial civilization a mistake! * I'm not blindly against regulation. I think all laws can and should be viewed as forms of regulations, and I don't think it's feasible for society to exist without laws. * That said, I'm also not blindly in favor of regulation, even for AI risk. You have to show me that the benefits outweigh the harm * I am generally in favor of thoughtful, targeted AI regulations that align incentives well, and reduce downside risks without completely stifling innovation. * I'm open to extreme regulations and policies if or when an AI catastrophe seems imminent, but I don't think we're in such a world right now. I'm not persuaded by the arguments that people have given for this thesis, such as Eliezer Yudkowsky's AGI ruin post.
One thing the AI Pause Debate Week has made salient to me: there appears to be a mismatch between the kind of slowing that on-the-ground AI policy folks talk about, versus the type that AI policy researchers and technical alignment people talk about. My impression from talking to policy folks who are in or close to government—admittedly a sample of only five or so—is that the main[1] coordination problem for reducing AI x-risk is about ensuring the so-called alignment tax gets paid (i.e., ensuring that all the big labs put some time/money/effort into safety, and that none “defect” by skimping on safety to jump ahead on capabilities). This seems to rest on the assumption that the alignment tax is a coherent notion and that technical alignment people are somewhat on track to pay this tax. On the other hand, my impression is that technical alignment people, and AI policy researchers at EA-oriented orgs,[2] are not at all confident in there being a viable level of time/money/effort that will produce safe AGI on the default trajectory. The type of policy action that’s needed, so they seem to say, is much more drastic. For example, something in the vein of global coordination to slow, limit, or outright stop development and deployment of AI capabilities (see, e.g., Larsen’s,[3] Bensinger’s, and Stein-Perlman’s debate week posts), whilst alignment researchers scramble to figure out how on earth to align frontier systems. I’m concerned by this mismatch. It would appear that the game plans of two adjacent clusters of people working to reduce AI x-risk are at odds. (Clearly, this is an oversimplification and there are a range of takes from within both clusters, but my current epistemic status is that this oversimplification gestures at a true and important pattern.) Am I simply mistaken about there being a mismatch here? If not, is anyone working to remedy the situation? Or does anyone have thoughts on how this arose, how it could be rectified, or how to prevent similar mismatches from arising in the future? 1. ^ In the USA, this main is served with a hearty side order of “Let’s make sure China in particular never gets ahead on capabilities.” 2. ^ e.g., Rethink Priorities, AI Impacts 3. ^ I’m aware that Larsen recently crossed over into writing policy bills, but I’m counting him as a technical person on account of his technical background and his time spent in the Berkeley sphere of technical alignment people. Nonetheless, perhaps crossovers like this are a good omen for policy and technical people getting onto the same page.
I'm fairly disappointed with how much discussion I've seen recently that either doesn't bother to engage with ways in which the poster might be wrong, or only engages with weak versions. It's possible that the "debate" format of the last week has made this worse, though not all of the things I've seen were directly part of that. I think that not engaging at all, and merely presenting one side while saying that's what you're doing, seems better than presenting and responding to counterarguments (but only the weak ones), which still seems better than strawmanning arguments that someone else has presented.
Who tends to be clean? With all the scandals in the last year or two, has anyone looked at which recruitment sources are least likely to produce someone extremely net negative in direct impact or to the community (i.e. a justified scandal)? Maybe this should inform outreach efforts.

Recent discussion

tl;dr An indefinite AI pause is a somewhat plausible outcome and could be made more likely if EAs actively push for a generic pause. I think an indefinite pause proposal is substantially worse than a brief pause proposal, and would probably be net negative. I recommend that alternative policies with greater effectiveness and fewer downsides should be considered instead.

Broadly speaking, there seem to be two types of moratoriums on technologies: (1) moratoriums that are quickly lifted, and (2) moratoriums that are later codified into law as indefinite bans. 

In the first category, we find the voluntary 1974 moratorium on recombinant DNA research, the 2014 moratorium on gain of function research, and the FDA’s partial 2013 moratorium on genetic screening.

In the second category, we find the 1958 moratorium on conducting nuclear tests above the...

I think that as systems get more capable, we will see a large increase in our alignment efforts and monitoring of AI systems, even without any further intervention from longtermists.

Maybe so. But I can't really see mechanistic interpretability being solved to a sufficient degree to detect an AI playing the training game, in time to avert doom. Not without a long pause first at least!

See the reply to the first comment on that post. Paul's "most humans die from AI takeover" is 11%. There are other bad scenarios he considers, like losing control of the future, or most humans die for other reasons, but my understanding is that the 11% most closely corresponds to doom from AI.
I'm surprised by your 25%. To me, that really doesn't match up with  from your essay.

I'm sure this is a very unpopular take but I feel obliged to share it: I find the "pausing AI development is impossible" arguments extremely parallel to the "economic degrowth in rich countries is impossible" arguments; and the worse consequences for humanity (and its probabilities) of not doing doing them not too dissimilar. I find it baffling (and epistemically bad) how differently these debates are treated within EA. 

Although parallel arguments can be given for and against both issues, EA have disregarded the possibility to degrowth the economy in rich countries without engaging the arguments. Note that degrowthers have good reasons to believe that continued economic growth would lead to ecological collapse --which could be considered an existential risk as, although it would clearly not lead to the...

This is a copy of the English version a statement released yesterday by a group of academics that can be seen at The Spanish translation, by Mónica A. Ulloa Ruiz, will be put on the forum soon

This statement was drawn up by a group of researchers from a variety of institutions who attended the FHI and CSER Workshop on Pluralisms in Existential Risk Studies from 11th-14th May 2023. It conveys our support for the necessity for the community concerned with existential risk to be pluralistic, containing a diversity of methods, approaches and perspectives, that can foster difference and disagreement in a constructive manner. We recognise that the field has not yet achieved this necessary pluralism, and commit to bring about such pluralism. A list of researchers...

Just a thought here. I am not sure if you can literally read this as EA being overwhelmingly left, as it depends a lot on your view point and what you define as "left". EA exists both in the US and Europe. Policy positions that are seen as left and especially center left in the US would often be more on the center or center right spectrum in Europe.

Sign up for the Forum's email digest
You'll get a weekly email with the best posts from the past week. The Forum team selects the posts to feature based on personal preference and Forum popularity, and also adds some announcements and a classic post.

Join each Sunday to talk with Christians about Effective Altruism! (2 PM NYC time, 7PM London time)

All are welcome.

This week we discuss: Economic Growth as an EA Cause

This week we discuss a common theme in the development literature and also a common critique of EA: the importance of economic growth, which arguably dwarfs the significance of one-off interventions. Do we know what causes growth? How can we find out?

We meet first over Zoom: .

The first 15 minutes are introductions and announcements, followed by a 10-15 minute intro to the topic, followed by 30 minutes of breakout room discussions.

After about 1 hour, we may move over to gather for more friendly discussions / hangout.

Note: This post contains personal opinions that don’t necessarily match the views of others at CAIP

Executive Summary

Advanced AI has the potential to cause an existential catastrophe. In this essay, I outline some policy ideas which could help mitigate this risk. Importantly, even though I focus on catastrophic risk here, there are many other reasons to ensure responsible AI development. 

I am not advocating for a pause right now. If we had a pause, I think it would only be useful insofar as we use the pause to implement governance structures that mitigate risk after the pause has ended. 

This essay outlines the important elements I think a good governance structure would include: visibility into AI development, and brakes that the government could use to stop dangerous AIs from being built. 

First, I’ll summarize some claims...

I'll be looking forward to hearing more about your work on whistleblowing! I've heard some promising takes about this direction. Strikes me as broadly good and currently neglected.


This post was originally intended as a follow-up post to Josh’s Are short timelines actually bad?, but given the AI pause debate, I’ve adapted it slightly and forced myself to get it to a readable form, it’s still a draft. 

In terms of the debate, this post is relevant because I think people often believe that it is better if AGI development happens later compared to sooner (hope AI timelines are long). I used to believe this, and now I think it’s incredibly unclear and we should be very uncertain. Josh’s post covers some arguments for why acceleration may be good: Avoid/delay a race with ChinaSmooth out takeoff (reduce overhangs), Keep the good guys in the lead. In this post I discuss two other arguments that might point toward acceleration: AGI development centralization...

Meta’s frontier AI models are fundamentally unsafe. Since Meta AI has released the model weights publicly, any safety measures can be removed. Before it releases even more advanced models – which will have more dangerous capabilities – we call on Meta to take responsible release seriously and stop irreversible proliferation. Join us for a peaceful protest at Meta’s office in San Francisco at 250 Howard St at 4pm PT.

RSVP on Facebook[1] or through this form.

Let’s send a message to Meta:

  • Stop irreversible proliferation of model weights. Meta’s models are not safe if anyone can remove the safety measures.
  • Take AI risks seriously.
  • Take responsibility for harms caused by your AIs.
  • Stop free-riding on the goodwill of the open-source community. Llama models are not and have never been open source, says the Open
I think the protests could benefit from "gears-level" detail on the sort of proximate effect as well as medium-long term effects: 1. What literally do people do in a protest, how should they dress, speak, what impression should they induce in their audience and what is their audience? 2. What could the medium or long term effects of the protest be? E.g. press, online discussion, awareness, sentiment I think there is a good understanding here based on the comments from the OP below.  Note that this doesn't have to be airtight or 100% accurate. But this would help discourse and understanding. I'm slightly worried/skeptical that the discourse around them isn't high-quality, e.g. many of these comments are speculative expressions of personal feelings.

I agree, it seems like there is a pretty big knowledge gap here on protests, more than I had thought. I’ll bump stirring a doc like this up in priority.

Great to see you getting behind this Matthew. Being against the open sourcing of frontier AI is something that we can have a broader coalition on.

in general I think it's much easier for people to do great research and actually figure stuff out when they're viscerally interested in the problems they're tackling, and excited about the process of doing that work.

Totally. But OP kinda made it sound like the fact that you found 2 depressing was evidence it was the wrong direction. I think advocacy could be fun and full of its own fascinating logistical and intellectual questions as well as lots of satisfying hands-on work.

Obviously if individual people want to use or not use a given product, that's their business. I'm calling it out not as a criticism of individuals, but in the context of setting the broader AI safety culture, for two broad reasons: 1. In a few years' time, the ability to use AIs will be one of the strongest drivers of productivity, and not using them will be... actually, less Luddite, and more Amish. It's fine for some people to be Amish, but for AI safety people (whose work particularly depends on understanding AI well) not using cutting-edge AI is like trying to be part of the original hacker culture while not using computers.  2. I think that the idea of actually trying to do good effectively is a pretty radical one, and scope-sensitivity is a key component of that. Without it, people very easily slide into focusing on virtue signalling or ingroup/outgroup signalling (e.g. climate activists refusing to take flights/use plastic bags/etc), which then has knock-on effects in who is attracted to the movement, etc. On twitter I recently criticized a UK campaign to ban a specific dog breed for not being very scope-sensitive; you can think of this as similar to that.
Yepp, I disagree on a bunch of counts. a) I dislike the phrase "we all die", nobody has justifiable confidence high enough to make that claim, even if ASI is misaligned enough to seize power there's a pretty wide range of options for the future of humans, including some really good ones (just like there's a pretty wide range of options for the future of gorillas, if humans remain in charge). b) Same for "the capability program is an easier technical problem than the alignment program". You don't know that; nobody knows that; Lord Kelvin/Einstein/Ehrlich/etc would all have said "X is an easier technical problem than flight/nuclear energy/feeding the world/etc" for a wide range of X, a few years before each of those actually happened. c) The distinction between capabilities and alignment is a useful concept when choosing research on an individual level; but it's far from robust enough to be a good organizing principle on a societal level. There is a lot of disagreement about what qualifies as which, and to which extent, even within the safety community; I think there are a whole bunch of predictable failure modes of the political position that "here is the bad thing that must be prevented at all costs, and here is the good thing we're crucially trying to promote, and also everyone disagrees on where the line between them is and they're done by many of the same people". This feels like a recipe for unproductive or counterproductive advocacy, corrupt institutions, etc. If alignment researchers had to demonstrate that their work had no capabilities externalities, they'd never get anything done (just as, if renewables researchers had to demonstrate that their research didn't involve emitting any carbon, they'd never get anything done). I will write about possible alternative framings in an upcoming post. As written, I would oppose this. I doubt the world as a whole could solve alignment with zero AI experiments; feels like asking medieval theologians to figure out the

This is a collection of resources that I recommend for how, and why, to pursue a career in animal advocacy - particularly if you think animal advocacy might not be for you.

At EAGx Australia 2023, I'm giving a lightning talk on why and how to pursue a career in animal advocacy. I thought I'd make an EA Forum post with links to everything I talk about, as it may help to have all of these resources in one place. The slides from my lightning talk are available here.

Launching independent projects in animal advocacy

  • Charity Entrepreneurship incubation programme
    • Applications open until September 30
    • Please apply!
    • Link
  • How to Launch a High-Impact Nonprofit, handbook by Charity Entrepreneurship (link)
    • I can confirm this is a truly excellent resource
  • "Who Is a Good Fit for a Career in

This is a great list of resources! One thing I'd add is that the effective animal advocacy space is pretty seriously funding constrained right now, and I don't see any signs that the situation is likely to change in the next few years. For that reason, I think it's worth calling out earning to give as a potentially uniquely promising path to impact. Animal Advocacy Careers had a good post on ETG for animals a few months ago.