So, a while back I came up with an obscure idea I called the Alpha Omega Theorem and posted it on the Less Wrong forums. Given how there's only one post about it, it shouldn't be something that LLMs would know about. So in the past, I'd ask them "What is the Alpha Omega Theorem?", and they'd always make up some nonsense about a mathematical theory that doesn't actually exist. More recently, Google Gemini and Microsoft Bing Chat would use search to find my post and use that as the basis for their explanation. However, I only have the free version of ChatGPT and Claude, so they don't have access to the Internet and would make stuff up.

A couple days ago I tried the question on ChatGPT again, and GPT-4o managed to correctly say that there isn't a widely known concept of that name in math or science, and basically said it didn't know. Claude still makes up a nonsensical math theory. I also today tried telling Google Gemini not to use search, and it also said it did not know rather than making stuff up.

I'm actually pretty surprised by this. Looks like OpenAI and Google figured out how to reduce hallucinations somehow.

Just wanted to point out that the distinction between total and average utilitarianism predates Derek Parfit's Reasons and Persons, with Henry Sidgwick discussing it in the Methods of Ethics from 1874, and John Harsanyi advocates for a form of average utilitarianism in Morality and the Theory of Rational Behaviour from 1977.

Other than that, great post! I feel for your moral uncertainty and anxiety. It reminds me of the discussions we used to have on the old Felicifia forums back when they were still around. A lot of negative leaning utilitarians on Felicifia actually talked about hypothetically ending the universe to end suffering and such, and the hedonium shockwave thing was also discussed a fair bit, as well as the neuron count as sentience metric proxy thing.

A number of Felicifia alumni later became somewhat prominent EAs like Brian Tomasik and Peter Wildeford (back then he was still Peter Hurford).

I'm wondering what people's opinions are on how urgent alignment work is. I'm a former ML scientist who previously worked at Maluuba and Huawei Canada, but switched industries into game development, at least in part to avoid contributing to AI capabilities research. I tried earlier to interview with FAR and Generally Intelligent, but didn't get in. I've also done some cursory independent AI safety research in interpretability and game theoretic ideas my spare time, though nothing interesting enough to publish yet.

My wife also recently had a baby, and caring for him is a substantial time sink, especially for the next year until daycare starts. Is it worth considering things like hiring a nanny, if it'll free me up to actually do more AI safety research? I'm uncertain if I can realistically contribute to the field, but I also feel like AGI could potentially be coming very soon, and maybe I should make the effort just in case it makes some meaningful difference.

Though 80% of EAs are left-leaning, the plurality of about 40% of EAs are centre-left, and only 37% classified themselves as left. Most likely the centre-left are social liberals and left-leaning centrists, while a good chunk of the left are social democrats and progressives. Actual leftists in the sense of democratic socialists/anarchists/Marxists/etc. are probably much less than that. 5 out of 21 is about 24%, which would be 65% of the 37% that are generally left-wing.

Source: (Survey Data)

This does not bode well to me. One of my personal concerns about the usefulness of AI safety technical research is the extent to which the fruits of such research would actually be utilized by the frontier labs in practice. Just because some hypothetical researcher or lab figures out a solution to the Alignment problem, it doesn't mean the actual eventual creators of AGI will care enough to actually use it if it, for instance, comes with an alignment tax that slows down their capabilities work and leads to less profit, or worse, causes the loss of first mover advantage to a less scrupulous competitor.

OpenAI seems like the front runner right now, and the fact they had a substantial Alignment Team with substantial compute resources devoted to them, at least made it seem like maybe they'd care enough to use any effective alignment techniques that do get developed and ensure that things go well. The gutting of the Alignment Team does not look good in this regard.

I also want to point out that many of the EAs sympathetic to socialism might have better judgment than people like me and not want to out themselves publicly on the forums, just in case it might affect future chances at things like funding. We'd like to think that it shouldn't matter, but it may just be prudent not to take the risk.

Out of 35 comments so far I count 21 commenters, with 5 sounding particularly sympathetic to socialism, in my subjective view. Most of the rest seem neutral or ambivalent rather than strongly opposed, and the 2 eugenicists have both been severely downvoted.

Relevant XKCD comic.

To further comment, this seems like it might be an intractable task, as the term "dependency hell" kind of implies. You'd have to scrap likely all of GitHub and calculate what libraries are used most frequently in all projects to get an accurate assessment. Then it's not clear to me how you'd identify their level of resourcing. Number of contributors? Frequency of commits?

Also, with your example of the XZ attack, it's not even clear who made the attack. If you suspect it was, say, the NSA, would you want to thwart them if their purpose was to protect American interests? (I'm assuming you're pro-American) Things like zero-days are frequently used by various state actors, and it's a morally grey question whether or not those uses are justified.

I also, as a comp sci and programmer, have doubts you'd ever be able to 100% prevent the risk of zero-days or something like the XZ attack from happening in open source code. Given how common zero-days seem to be, I suspect there are many in existing open source work that still haven't been discovered, and that XZ was just a rare exception where someone was caught. 

Yes, hardening these systems might somewhat mitigate the risk, but I wouldn't know how to evaluate how effective such an intervention would be, or even, how you'd harden them exactly. Even if you identify the at-risk projects, you'd need to do something about them. Would you hire software engineers to shore up the weaker projects? Given the cost of competent SWEs these days, that seems potentially expensive, and could compete for funding with actual AI safety work.

As an EA who has been in the movement since 2013 and a self-proclaimed liberal democratic socialist, I'd say that there is definitely a tension between EA and socialism that stems at least in part from the history of both movements.

One of the basic foundations of EA thought is Utilitarianism, and historically, Marx criticized Bentham and Mill for what he considered "bourgeois morality" that merely justified the rule of the ruling class. Utilitarianism's influence on EA can be traced to Peter Singer and also the Oxford moral philosophy students turned professors like Will MacAskill and Toby Ord. That's what I'd consider the academic foundation of EA, and one of four major power centres in EA.

The other three power centres are, respectively:

• The Bay Area Rationalist community, led more or less by Eliezer Yudkowsky (who early on was funded by FHI at Oxford to start his blogging), and who are known for having something of a techno-libertarian bias.
• The billionaire Dustin Moskovitz through Open Philanthropy, who funds a massive percentage of EA related projects (nothing against him personally, but the optics are clearly challenging).
• The DC Area American establishment, including think tanks like RAND (who's current leader is a known EA supporter), although it's hard to say to what extent EA is trying to influence the establishment vs. the other way around, but there's probably significant cross-pollenization, especially more recently with the AI governance push.

All of these would be considered suspect by most card-carrying socialists, particularly the more radical ones who are prone to disliking an Anglo-centric movement beholden to both political and entrepreneurial elites.

A more radical socialist (i.e. tankies) might even extend the known history of CIA and U.S. government PSYOPs into a conspiracy theory that EA is a possible PSYOP to create a funnel for would be left-leaning radical students to be safely redirected into a relatively tame, American Imperialism conforming ideology that doesn't aspire to upset the status quo in any meaningful way. While I doubt this would be anything more than a silly conspiracy theory, socialists who have dealt with a long history of red scare tactics and government surveillance are likely to fall prey to this kind of paranoia.

All that, before I even got to the ideological clashes of EA and socialism.

Ideologically, EA, particularly the leadership of EA, is very much biased towards western liberalism, both the intellectual tradition, and the political movement. Bentham and Mill were both considered liberals in their day (notwithstanding Bentham's connections to Robert Owen, or Mill's later turn towards cooperatives). Oxford's elites today seem generally more aligned with liberal thinking than socialist thinking, the Bay Area folks lean libertarian (i.e. classical liberalism), and of course the American establishment is very much a defender of the liberalism of Fukuyama's End of History.

The idea for instance, of private charitable donations to GiveWell approved charities to administer bednets for a health intervention in Africa is something that makes the most sense if you are a liberal with individualist sensibilities. Socialists would almost certainly ask, why isn't this health intervention being done by the government? Shouldn't that be the responsibility of the state or society to provide for the basic welfare of its citizens?

This is not to say that EA and Socialism have no common ground. Your post shows clearly that there are places of overlap, particularly in the ideal of some form of altruism being desirable. The rank and file EA, according to surveys, is most likely to lean centre-left to left on the political spectrum, and likely at least somewhat sympathetic to the idealism of socialism, if not necessarily its practice.

I don't think the difficulties are insurmountable, but it would probably require a substantial push for engagement from the rank and file left-leaning EAs that would somehow be listened to by the EA leadership rather than being briefly considered and then ultimately ignored. If you haven't guessed, I'm somewhat cynical about the EA leadership, and doubtful that they'd do this, given that the power centres I've mentioned hold considerable sway.

Good luck though!

So, I have three very distinct ideas for projects that I'm thinking of applying for funding from the Long Term Future Fund for. I was wondering if anyone knows whether it makes more sense to submit them together in one application, or as three separate individual applications?