Quick points:
1. I've come to believe that work in foundational political change is fairly neglected, in-comparison to its value.
2. As Scott Alexander wrote, political donations are surprisingly small for their impact. This seems especially true for someone as radical as Trump.
3. Related, the upper-class has been doing fantastically these last 10-30 years or so, and now has a very large amount of basically-spare capital. 
4. I very much expect that there could be arrangements that are positive-EV to groups of these wealthy individuals, to help us have better political institutions. 

So a corresponding $10T+ question is, "How to we set up structures whereby spare capital (which clearly exists) gets funneled into mutually-beneficial efforts to improve governments (or other similar institutions)"

A very simple example would be something like, "GiveWell for Political Reform." (I know small versions of this have been tried. Also, I know it would be very tough to find ways to get people with spare capital to part with said capital.)

I wrote one specific futuristic proposal here. I expect that better epistemics/thinking abilities will help a lot here. I'm personally working on epistemic improvements, in large part to help with things like this. 

If you've ever written or interacted with Squiggle code before, we at QURI would really appreciate it if you could fill out our Squiggle Survey!

https://docs.google.com/forms/d/e/1FAIpQLSfSnuKoUUQm4j3HEoqPmTYiWby9To8XXN5pDLlr95AiKa2srg/viewform

We don't have many ways to gauge or evaluate how people interact with our tools. Responses here will go a long way to deciding on our future plans.

Also, if we get enough responses, we'd like to make a public post about ways that people are (and aren't) using Squiggle. 

That roughly sounds right to me. 

I think that power/incentives often come first, then organizations and ecosystems shape their epidemics to some degree in order to be convenient. This makes it quite difficult what causally led to what. 

At the same time, I'm similarly suspicious of a lot of epistemics. It's obviously not just beliefs that OP likes that will be biased to favor convenience. Arguably a lot of these beliefs just replace other bad beliefs that were biased to favor other potential stakeholders or other bad incentives. 

Generally I'm quite happy for people and institutions to be quite suspicious of their worldviews and beliefs, especially ones that are incentivized by their surroundings. 

(I previously wrote about some of this in my conveniences post here, though that post didn't get much attention.)

Instead of "Goodharting", I like the potential names "Positive Alignment" and "Negative Alignment."

"Positive Alignment" means that the motivated party changes their actions in ways the incentive creator likes. "Negative Alignment" means the opposite.

Whenever there are incentives offered to certain people/agents, there are likely to be cases of both Positive Alignment and Negative Alignment. The net effect will likely be either positive or negative. 

"Goodharting" is fairly vague and typically just refers to just the "Negative Alignment" portion. 

I'd expect this to make some discussion clearer.
"Will this new incentive be goodharted?" -> "Will this incentive lead to Net-Negative Alignment?" 

Other Name Options

Claude 3.7 recommended other naming ideas like:

• Intentional vs Perverse Responses
• Convergent vs Divergent Optimization
• True-Goal vs Proxy-Goal Alignment
• Productive vs Counterproductive Compliance

This context is useful, thanks.

Looking back, I think this part of my first comment was poorly worded:
> I imagine that scientists will soon have the ability to be unusually transparent and provide incredibly low rates of fraud/bias, using AI.

I meant 
> I imagine that scientists will [soon have the ability to] be unusually transparent and provide incredibly low rates of fraud/bias], using AI.

So it's not that this will lead to low rates of fraud/bias, but that AI will help enable that for scientists willing to go along with it - but at the same time, there's a separate question of if scientists are willing to go along with it. 

But I think even that probably is not fair. A a better description of my beliefs is something like,

• I think that LLM auditing tools could be useful for some kinds of scientific research for communities open to them.
• I think in the short-term, sufficiently-motivated groups could develop these tools and use them to help decrease the levels of statistical and algorithmic accidents that happen. Correspondingly, I'd expect this to help with fraud. 
• In the long-run, whenever AI approaches human-level intelligence (which I think will likely happen in the next 20 years, but I realize others disagree), I expect that more and more of the scientific process will be automated. I think there are ways this could go very well using things like AI auditing, whereby the results will be much more reliable than those currently made by humans. There are of course also worlds in which humans do dumb things with the AIs and the opposite happens. 
• I think that at least, AI safety researchers should consider using these kinds of methods, and that the AI safety landscape should investigate efforts to make decent auditing tools."

My core hope with the original message is to draw attention to AI science auditing tools as things that might be interesting/useful, not to claim that they're definitely a major game changer. 

I think this is a significant issue, though I imagine a lot of this can be explained more by the fact that OP is powerful than that it is respected. 

If your organization is highly reliant on one funder, then doing things that funder regards as good is a major factor that will determine if you will continue to get funding, even if you might disagree. So it could make a lot of sense to update your actions towards that funder, more than would be the case if you had all the power.

I think that decentralizing funding is good insofar as the nonprofit gets either more power (to the extent that this is good) or better incentives. There are definitely options where one could get more funding, but that funding could come from worse funders, and then incentives decline.

Ultimately, I'd hope that OP and other existing funders can improve, and/or we get other really high-quality funders. 

Okay, the winner has been announced! It's dmartin80 on LessWrong. More here:

https://www.lesswrong.com/posts/AA8GJ7Qc6ndBtJxv7/usd300-fermi-model-competition?commentId=v58BLaEA7KzRi3kmS

This strikes comment strikes me as so different to my view that I imagine you might be envisioning a very specific implementation of AI auditors that I'm not advocating for. 

I tried having a discussion with an LLM about this to get some more insight, you can see this here if you like (though I suspect that you won't wind this useful, as you seem to not trust LLMs much at all.) It wound up suggesting implementations that could still provide benefits while minimizing potential costs.

https://claude.ai/share/4943d5aa-ed91-4b3a-af39-bc4cde9b65ef

The bigger issue here is with the "auditors" themselves: who's in charge of them? If a working scientist disagrees with what the "auditor" says, what happens? What happens if someone like Elon is in charge, and decides to use the auditors for a political crusade against "woke science", as is currently literally happening right now? 

I think this is a very sensible question.

My obvious answer is that the auditors should be held up to higher standards than the things they are auditing. This means that these should be particularly open, and should be open to other auditing. For example, the auditing code could be open-source, highly tested, and evaluated by both humans and AI systems. 

I agree that there are ways one could do a poor job with auditing. I think this is generally true for most powerful tools we can bring in - we need to be sure to use it well, else it could do harm.

On your other points - it sounds like you have dramatically lower expectations for AI than I do or much of the AI safety community does. I agree that if you don't think AI is very exciting, then AI-assisted auditing probably won't go that far. 

From my post:
> this could all be good experimentation on our way to systems that will oversee key AI progress. I ultimately want AI auditors for all risky AI development, but some of that will be a harder sell.

If it's the case that AI-auditors won't work, then I assume we wouldn't particularly need to oversee key AI progress anyway, as there's not much to oversee. 

I wasn't trying to make the argument that it would definitely be clear when this window closes. I'm very unsure of this. I also expect that different people have different beliefs, and that it makes sense for them to then take corresponding actions.