Epistemic status: Speculative fiction. This is lightly adapted from an earlier shortform.
 

It's difficult to imagine how human epistemics and AI will play out. On one hand, AI could provide much better information and general intellect. On the other hand, AI could help people with incorrect beliefs preserve those false beliefs indefinitely. 

Will advanced AIs attempting to rationalize bad beliefs be able to outmatch AIs providing good ones?

While I think that some AI systems could do fantastic things for human epistemics, I'm also worried about lock-in scenarios where people fall into self-reinforcing cycles overseen by AIs. It's possible that a great deal of lock-in might happen in the next 30 years or so (if you believe AGI/TAI might happen soon), so this could be something to take seriously.

While it might be easy to imagine extremes on either end of this, I expect that the future will feature a mix of positives and negatives, and that future epistemic tensions will mirror previous ones.

Here's one incredibly rough outline of one potential future I could envision, as an example. This example assumes that humanity broadly gets AI alignment right.

It's 2028.

MAGA types typically use DeepReasoning-MAGA, or DR-MAGA. The far left typically uses DR-JUSTICE. People in the middle often use DR-INTELLECT, which has the biases and worldview of a somewhat normal citizen.

Some niche technical academics (the same ones who currently favor Bayesian statistics) and hedge funds use DR-BAYSIAN or DRB for short. DRB is known to have higher accuracy than the other models, but gets a lot of public hate for having controversial viewpoints. It's also fairly slow and expensive, so a poor fit for large-scale use. DRB is known to be fairly off-putting to chat with and doesn't get much promotion.

Bain and McKinsey both have their own offerings, called DR-Bain and DR-McKinsey, respectively. These are a bit like DR-BAYSIAN, but are munch punchier and confident. They're highly marketed to managers. These tools produce really fancy graphics, and specialize in things like not leaking information, minimizing corporate decision liability, being easy to use by old people, and being customizable to represent the views of specific companies.

For a while now, some evaluations produced by intellectuals have demonstrated that DR-BAYSIAN seems to be the most accurate, but few others really care or notice this. DR-MAGA has figured out particularly great techniques to get users to distrust DR-BAYSIAN.

Betting gets weird. Rather than making specific bets on specific things, users started to make meta-bets. "I'll give money to DR-MAGA to bet on my behalf. It will then make bets with DR-BAYSIAN, which is funded by its believers."

At first, DR-BAYSIAN dominates the bets, and its advocates earn a decent amount of money. But as time passes, this discrepancy diminishes. A few things happen:

1. All DR agents converge on beliefs over particularly near-term and precise facts.
2. Non-competitive betting agents develop alternative worldviews in which these bets are invalid or unimportant.
3. Non-competitive betting agents develop alternative worldviews that are exceedingly difficult to empirically test.

In many areas, items 1-3 push people to believe more in the direction of the truth. Because of (1), many short-term decisions get to be highly optimal and predictable.

But because of (2) and (3), epistemic paths diverge, and non-betting-competitive agents get increasingly sophisticated at achieving epistemic lock-in with their users.

Some DR agents correctly identify the game theory dynamics of epistemic lock-in, and this kickstarts a race to gain converts. It seems like advent users of DR-MAGA are very locked-down in these views, and forecasts don't see them ever changing. But there's a decent population that isn't yet highly invested in any cluster. Money spent convincing the not-yet-sure goes a much further way than money spent convincing the highly dedicated, so the cluster of non-deep-believers gets highly targeted for a while. It's basically a religious race to gain the remaining agnostics.

At some point, most people (especially those with significant resources) are highly locked in to one specific reasoning agent.

After this, the future seems fairly predictable again. TAI comes, and people with resources broadly gain correspondingly more resources. People defer more and more to the AI systems, which are now in highly stable self-reinforcing feedback loops.

Coalitions of people behind each reasoning agent delegate their resources to said agents, then these agents make trades with each other. The broad strokes of what to do with the rest of the lightcone are fairly straightforward. There's a somewhat simple strategy of resource acquisition and intelligence enhancement, followed by a period of exploiting said resources. The specific exploitation strategy depends heavily on the specific reasoning agent cluster each segment of resources belongs to.

Reflecting on this, several questions come to mind.

1. How much of an advantage will more honest/correct AI systems have in the future, when it comes to convincing people of things, particularly of things critical to epistemic lock-in?
2. How possible is it for AI systems with strong epistemics to be unpopular? More specifically - what aspects of epistemics should we expect AI labs to optimize, and which should we expect to be overlooked or intentionally done poorly?
3. Do we expect such a epistemic lock-in to happen, around TAI? If so, this would imply that it could be worth a lot of investment to try to improve epistemics quickly.
4. Where is the line between values and epistemics? I think that "epistemic lock-in" is a bigger deal than "value lock-in" or similar, but that's much because I expect that epistemics change values more than values change epistemics. There's been previous discussion around effective altruism of "value lock-in," and from what I can tell, very little of "epistemic lock-in." I suspect this disparity is a mistake.
5. What will happen regarding epistemic clusters and government? What about AI labs? There are probably a few actors here who particularly matter.