AI's are already getting money with crypto memecoins. Wondering if there might be some kind of unholy mix of AI generated memecoins, crypto ransomware and self-replicating AI viruses unleashed in the near future.

Re "role-playing", that is moot when it's the end result that matters - what actions the AI takes in the world. See also: Frontier AI systems have surpassed the self-replicating red line. 

Shutdown Avoidance ("do self-replication before being killed"), combined with the recent Apollo o1 research on propensity to attempt self-exfiltration, pretty much closes the loop on misaligned AIs escaping when given sufficient scaffolding to do so.

One can hope that the damage is limited and that it serves as an appropriate wake-up call to governments. I guess we'll see..

Surely it's just a matter of time -- now that the method has been published -- before AI models are spreading like viruses?

This is brilliant. I agree with almost all of it^[1] - it's a good articulation of how my own thinking on this has evolved over the last couple of years^[2]. My timelines might be shorter, and my p(doom) higher, but it's good to see an exposition for how one need not have such short timelines or high p(doom) to still draw the same conclusions. I recently donated significant amounts to PauseAI Global and PauseAI US. Your $30k to PauseAI US will get them to 5/6 of their current fundraising target - thank you!

1. ^{^}

   Some points of disagreement, additional information, and emphasis in other comments I made as I read through.

2. ^{^}

   Actually to be fair, it's more detailed!

You should talk to David Pearce. His view of physicalism (phenomenal binding) precludes consciousness in digital minds^[1].

1. ^{^}

   But he also goes further and claims that world-ending ASI is impossible for the reason of it requiring, yet lacking, unitary conscious experience (whereas I think that a "blind idiot god" is perfectly capable of destroying everything we value).

I know of at least one potential counterexample: OpenAI’s RLHF was developed by AI safety people who joined OpenAI to promote safety. But it’s not clear that RLHF helps with x-risk.

I'd go further and say that it's not actually a counterexample. RLHF allowed OpenAI to be hugely profitable - without it they wouldn't've been able to publicly release their models and get their massive userbase.

(I can see an argument for blocking entrances to AI company offices, but I think the argument for blocking traffic is much weaker.)

I think Stop AI have taken this criticism onboard (having encountered it from a number of places). Their plan for the last couple of months has been to keep blocking OpenAI's gates until they have their day in court^[1] where they can make a "necessity" case for breaking the law to prevent a (much much) larger harm from occurring (or to prevent OpenAI from recklessly endangering everyone). Winning such a case would be huge.

1. ^{^}

   They feature heavily in this recent documentary that is well worth a watch.

I don't understand what's going on here psychologically—according to the expressed beliefs of people like Dario Amodei and Shane Legg, they're massively endangering their own lives in exchange for profit. It's not even that they disagree with me about key facts, they're just doing things that make no sense according to their own (expressed) beliefs.

Does anyone know what's going on here? Dan Fagella says it's a "Sardanapalus urge", to want to be destroyed by their sand god (not anyone else's), but I suspect it's something more like extreme hubris^[1] - irrational overconfidence. This is a very Silicon Valley / entrepreneurial trait. You pretty much have to go against the grain and against all the odds to win really big. But it's one thing with making money, and another with your life (and yet another with everyone else on the planet's lives too!). 

I strongly believe that if Amodei, Altman, Legg and Hassabis were sat round a table with Omega and a 6 shooter with even 1 bullet in the chamber, they wouldn't play a game of actual Russian Roulette with a prize of utopia/the glorious transhumanist future, let alone such a game with a prize of a mere trillion dollars.

1. ^{^}

   The biggest cases of hubris in the history of the known universe.