I find videos about space colonization pretty inspiring. Of course, space colonization would ideally be paired with some level of suffering abolition, so we aren't spreading needless suffering to other planets. Space colonization could help with political discord, since people with different ideas of a "good society" can band together and peacefully disperse through the solar system. If you think traveling the world to experience different cultures is fun, I expect visiting other planets to experience different cultures will be even better. On the AI front, rumor has it that scaling is slowing down... that could grant more time for alignment work, and increase the probability that an incredible future will come to pass.

I don't think OpenAI's near term ability to make money (e.g. because of the quality of its models) is particularly relevant now to its valuation. It's possible it won't be in the lead in the future, but I think OpenAI investors are betting on worlds where OpenAI does clearly "win", and the stickiness of its customers in other worlds doesn't really affect the valuation much.

They're losing billions every year, and they need a continuous flow of investment to pay the bills. Even if current OpenAI investors are focused on an extreme upside scenario, that doesn't mean they want unlimited exposure to OpenAI in their portfolio. Eventually OpenAI will find themselves talking to investors who care about moats, industry structure, profit and loss, etc.

The very fact that OpenAI has been throwing around revenue projections for the next 5 years suggests that investors care about those numbers.

I also think the extreme upside is not that compelling for OpenAI, due to their weird legal structure with capped profit and so on?

On the EA Forum it's common to think in terms of clear "wins", but it's unclear to me that typical AI investors are thinking this way. E.g. if they were, I would expect them to be more concerned about doom, and OpenAI's profit cap.

Dario Amodei's recent post was rather far out, and even in his fairly wild scenario, no clear "win" was implied or required. There's nothing in his post that implies LLM providers must be making outsized profits -- same way the fact that we're having this discussion online doesn't imply that typical dot-com bubble companies or telecom companies made outsized profits.

How much do you think customers having 0 friction to switching away from OpenAI would reduce its valuation? I think it wouldn't change it much, less than 10%.

If it becomes common knowledge that LLMs are bad businesses, and investor interest dries up, that could make the difference between OpenAI joining the ranks of FAANG at a $1T+ valuation vs raising a down round.

Markets are ruled by fear and greed. Too much doomer discourse inadvertently fuels "greed" sentiment by focusing on rapid capability gain scenarios. Arguably, doomer messaging to AI investors should be more like: "If OpenAI succeeds, you'll die. If it fails, you'll lose your shirt. Not a good bet either way."

There are liable to be tipping points here -- chipping in to keep OpenAI afloat is less attractive if future investors are seeming less willing to do this. There's also the background risk of a random recession due to H5N1 / a contested US election / port strike resumption / etc. to take into account, which could shift investor sentiment.

So I don't agree that working on this would be useful compared with things that contribute to safety more directly.

If you have a good way to contribute to safety, go for it. So far efforts to slow AI development haven't seemed very successful, and I think slowing AI development is an important and valuable thing to do. So it seems worth discussing alternatives to the current strategy there. I do think there's a fair amount of groupthink in EA.

a bet on OpenAI having better models in the future

OpenAI models will improve, and offerings from competitors will also improve. But will OpenAI's offerings consistently maintain a lead over competitors?

Here is an animation I found of LLM leaderboard rankings over time. It seems like OpenAI has consistently been in the lead, but its lead tends to be pretty narrow. They might even lose their lead in the future, given the recent talent exodus. [Edit: On the other hand, it's possible their best models are not publicly available.]

If switching costs were zero, it's easy for me to imagine businesses becoming price-sensitive. Imagine calling a wrapper API which automatically selects the cheapest LLM that (a) passes your test suite and (b) has a sufficiently low rate of confabulations/misbehavior/etc.

Given the choice of an expensive LLM with 112 IQ, and a cheap LLM with 110 IQ, a rational business might only pay for the 112 IQ LLM if they really need those additional 2 IQ points. Perhaps only a small fraction of business applications will fall in the narrow range where they can be done with 112 IQ but not 110 IQ. For other applications, you get commoditization.

A wrapper API might also employ some sort of router model that tries to figure out if it's worth paying extra for 2 more IQ points on a query-specific basis. For example, initially route to the cheapest LLM, and prompt that LLM really well, so it's good at complaining if it can't do the task. If it complains, retry with a more powerful LLM.

If the wrapper API was good enough, and everyone was using it, I could imagine a situation where even if your models consistently maintain a narrow lead, you barely eke out extra profits.

It's possible that https://openrouter.ai/ is already pretty close to what I'm describing. Maybe working there would be a good EA job?

[Idea to reduce investment in large training runs]

OpenAI is losing lots of money every year. They need continuous injections of investor cash to keep doing large training runs.

Investors will only invest in OpenAI if they expect to make a profit. They only expect to make a profit if OpenAI is able to charge more for their models than the cost of compute.

Two possible ways OpenAI can charge more than the cost of compute:

• Uniquely good models. This one's obvious.

• Switching costs. Even if OpenAI's models are just OK, if your AI application is already programmed to use OpenAI's API, you might not want to bother rewriting it.

Conclusion: If you want to reduce investment in large training runs, one way to do this would be to reduce switching costs for LLM users. Specifically, you could write a bunch of really slick open-source libraries (one for every major programming language) that abstract away details of OpenAI's API and make it super easy to drop in a competing product from Anthropic, Meta, etc. Ideally there would even be a method to abstract away various LLM-specific quirks related to prompts, confabulation, etc.

This pushes LLM companies closer to a world where they're competing purely on price, which reduces profits and makes them less attractive to investors.

The plan could backfire by accelerating commercial adoption of AI a little bit. My guess is that this effect wouldn't be terribly large.

There is this library, litellm. Seems like adoption is a bit lower than you might expect. It has ~13K stars on Github, whereas Django (venerable Python web framework that lets you abstract away your choice of database, among other things) has ~80K. So concrete actions might take the form of:

• Publicize litellm. Give talks about it, tweet about it, mention it on StackOverflow, etc. Since it uses the OpenAI format, it should be easy for existing OpenAI users to swap it in?

• Make improvements to litellm so it is more agnostic to LLM-specific quirks.

• You might even start a SaaS version of Perplexity.AI. Same way Perplexity abstracts away choice of LLM for the consumer, a SaaS version could abstract away choice of LLM for a business. Perhaps you could implement some TDD-for-prompts tooling. (Granted, I suppose this runs a greater risk of accelerating commercial AI adoption. On the other hand, micro-step TDD as described in that thread could also reduce demand for intelligence on the margin, by making it possible to get adequate results with lower-performing models.)

• Write libraries like litellm for languages besides Python.

I don't know if any EAs are still trying to break into ML engineering at this point, but if so I encourage them to look into this.

It encourages some form of "enlightened immobilism", where anyone proposing doing anything differently from the status quo gets instantly shut down.

I think telling people to critique less is a suboptimal solution for this. At least in theory, it's more ideal for people to be willing to do things despite getting critiques.

Someone can write a critique for anything. Instead of checking if there's a critique, you could check "does a neutral party think this critique is stronger than average for a random EA project" or something like that. (If the project is weaker than average in light of the critique, that suggests resources should perhaps be reallocated.)

Downside risk is everywhere, and its mere existence shouldn't be sufficient to cause inaction.

Deep learning is strongly biased toward networks that generalize the way humans want— otherwise, it wouldn’t be economically useful.

I noticed you switched here from talking about "SGD" to talking about "deep learning". That seems dodgy. I think you are neglecting the possible implicit regularization effect of SGD.

I don't work at OpenAI, but my prior is that insofar as ChatGPT generalizes, it's the result of many years of research into regularization being applied during its training. (The fact that the term 'regularization' doesn't even appear in this post seems like a big red flag.)

We've figured out now how to train neural networks so they generalize, and we could probably figure out how to train neural networks without schemers if we put in similar years of effort. But in the same way that the very earliest neural networks were (likely? I'm no historian) overfit by default, it seems reasonable to wonder if the very earliest neural networks large enough to have schemers will have schemers by default.

I understand that OpenAI's financial situation is not very good [edit: this may not be a high-quality source], and if they aren't able to convert to a for-profit, things will become even worse:

OpenAI has two years from the [current $6.6 billion funding round] deal’s close to convert to a for-profit company, or its funding will convert into debt at a 9% interest rate.

As an aside: how will OpenAI pay that interest in the event they can't convert to a for-profit business? Will they raise money to pay the interest rate? Will they get a loan?

https://www.wheresyoured.at/oai-business/

It's conceivable that OpenPhil suing OpenAI could buy us 10+ years of AI timeline, if the following dominoes fall:

• OpenPhil sues, and OpenAI fails to convert to a for-profit.

• As a result, OpenAI struggles to raise additional capital from investors.

• Losing $4-5 billion a year with little additional funding in sight, OpenAI is forced to make some tough financial decisions. They turn off the free version of ChatGPT, stop training new models, and cut salaries for employees. They're able to eke out some profit, but not much profit, because their product is not highly differentiated from other AI offerings.

• Silicon Valley herd mentality kicks in. OpenAI has been the hottest startup in the Valley. If it becomes known as the next WeWork, its fall will be earth-shaking. Game-theoretically, it doesn't make as much sense to invest in an early AI startup round if there's no capital willing to invest in subsequent rounds. OpenAI's collapse could generate the belief that AI startups will struggle to raise capital -- and if many investors believe that, it could therefore become true.

• The AI bubble deflates and the Valley refocuses on other industries.

It would be extremely ironic if the net effect of all Sam Altman and Mark Zuckerberg's efforts is to make AI companies uninvestable and buy us a bunch of timeline. Sam by generating a bunch of hype that fails to deliver, and Mark by commoditizing LLMs. (I say "ironic" because EAs are used to thinking of both Sam and Mark as irresponsible actors in the AI space.)

EDIT: there is some criticism of OpenPhil's approach to its public image here which may be relevant to the decision of whether to sue or not. Also, there's the obvious point that OpenAI appears to be one of the worst actors in the AI space.

EDIT 2: One also needs to consider how Sam might respond, e.g. by starting a new company and attempting to poach all OpenAI employees.

About a decade ago, I worked collecting signatures for ballot initiatives in California. I worked with a company which contracted with organizations that financially sponsored the ballot initiatives. At the time I was doing it, the sponsor would usually pay from between $1 to $4 per signature. I would stand in an area with lots of foot traffic and try to persuade passerby to sign my petitions. To maximize profits, the typical strategy is to order petitions from highest-paying to lowest-paying, and try to get any given passerby to sign as many petitions as possible. People who sign your petitions need to be registered voters, so if you're on e.g. a college campus with lots of people who aren't registered voters, you can carry voter registration forms in order to register them. But that strategy is more time-consuming and therefore less profitable. There is a process to randomly verify that petition signatories are registered voters to prevent fraud.

I got into this business because a friend of mine said it could be a good way to make extra cash if you find a good place to stand, and a good way to practice charisma. I got out of it because I wasn't making all that much money, and I was worried that bothering people who were just trying to go about their day was making me callous.

California's current petition system seems like a pretty clear perversion of whatever the designers had in mind. I barely remember having any sort of substantial discussion regarding the policy merits of the petitions I was collecting signatures for. The guy who ran the petition company freely admitted to collecting petitions for initiatives he didn't believe in, and said his most effective tactic for collecting a signatures was to emphasize that "this just puts it on the ballot". There was only one time I ever remember a lady who said "This sounds like bad policy so I'm not signing". I respected the heck out of that, even though her reasoning didn't persuade me. (I believe I was trying to collect her signature for the current top-paying petition related to minutiae of car insurance law.)

So overall, I agree that if you just want to put a petition on the ballot as efficiently as possible, and you have the money needed to hire contractors, then that's a good way to go. But I am also not terminally cynicism-pilled. Based on my insider knowledge of the system, I don't see any reason in principle why a group couldn't use the law more like it was intended.

Yes, collecting half a million signatures is a big project. Imagine 1000 volunteers, each committed enough to collect an average of 500 signatures each. But starting a mass movement is also a big project. So if you want to start a mass movement anyways, you might consider combining those objectives, and using contractors to make up any signature shortfall.

The AI Pause protests I've seen haven't struck me as very effective. I remember in the early days, some EAs were claiming that attending early Pause protests would be high impact, if the protests grew over time. Despite the poll numbers, the Pause protests don't seem to be growing much beyond the core EA/LW audience. If growing those protests is a goal, and a mass movement is considered desirable (a big "if", obviously), maybe it's time to embrace the grind and put in the same sort of leg work you see with e.g. vegan activism.

So -- I'm not suggesting volunteer signature collection as a substitute for professional signature collection, so much as I am suggesting volunteer signature collection as a substitute for doing protests. The Pause movement might see more growth if volunteers split off into groups of two and tried to talk to passerby about AI on an individual basis. AI is a hot topic, much hotter than car insurance law, and the hypothesis that passerby are interested in having 1-on-1 conversations about it may be worth cheap testing. Petitioning could serve as an excuse to start conversations which would ideally end in a signature, a mailing list signup, or a new committed volunteer.

I can share more strategy thoughts if people are interested.

To do a [citizen-initiated] ballot initiative, you stand on the street and ask passerby to sign your petition. Perhaps it would be possible to simultaneously build a mailing list of interested passerby to attend protests and such. That could translate the poll numbers into a stronger street presence.

I'm guessing Open Philanthropy would be well-positioned to sue, since they donated to the OpenAI non-profit.

Elon Musk is already suing but I'm not clear on the details: https://www.reuters.com/technology/elon-musk-revives-lawsuit-against-sam-altman-openai-nyt-reports-2024-08-05/ 

(Tagging some OpenAI staffers who might have opinions)

@JulianHazell @lukeprog @Jason Schukraft @Jasmine_Dhaliwal