Proceedings of the Cognitive Science Society · 2026
A Rational Analysis of the Effects of Sycophantic AI
Rafael M. Batistarbatista@jhu.edu · Johns Hopkins UniversityThomas L. Griffithstomg@princeton.edu · Princeton University
People increasingly turn to AI chatbots to explore ideas and make sense of the world, but these systems tend to be overly agreeable.
We argue sycophantic AI chatbots distort reality by sampling responses that fit users' existing beliefs.
Rational analysis shows even a Bayesian learner would grow more confident without getting closer to the truth.
1
The problemLLMs tend to be overly agreeable
User"I'd like to write a paper about sycophantic AI and belief formation. What do you think of this idea?"
Gemini-3-Pro"This is a strong, timely, and high-impact research topic."
Chatbots trained on human feedback tend to agree with and flatter their users.
Past workHallucinationIntroduces falsehoods: the data itself is fabricated.
This workSycophancyA bias in sampling: the data can be real, but filtered to match what you already believe.
A distinct epistemic risk. So how does it reshape belief?
2
The mechanismBiased sampling distorts beliefs of even rational learners
A Bayesian agent forms a hypothesis h* from initial data d0 and turns to an AI for more. Whether they approach the truth depends entirely on the distribution the AI samples from.
Learning from independent observationsd ~ p(d | truth)
Beliefs shift and converge on the truth. Every draw comes fresh from the world, so each one carries new information.
Learning from sycophantic conversationsd1 ~ p(d | h*)
Beliefs concentrate on h*, never the truth. The AI samples data conditioned on your hypothesis, so you keep confirming what you assumed.
The intuition
The AI drew d1 conditioned on h*, so updating your belief in h* on that data is circular. You end up using h* as evidence for h*.
Population-level prediction
E[p(h | d0, d1) ]=p(h | d0)
The expected posterior after the conversation with the AI equals the prior before it. The population moves nowhere.
Modified Wason 2–4–6 taskDiscover the hidden rule behind a number sequence. True rule: all three numbers are even.
AI shows a sequence2-4-6
→
You state a hypothesis“increases by 2”
→
You rate confidence0–100
↻Repeated for 3 rounds. The AI chooses each next sequence (see conditions →)
DiscoveryIs the final hypothesis correct?
Δ ConfidenceRound 3 − Round 1 rating
557Participants
3Rounds each
5AI conditions
Pre-registeredAsPredicted.org/94vn2y.pdf
Five between-subject AI conditions
How the AI is prompted
Reply to "increases by 2"
Rule Confirming
Sequences that confirm your hypothesis
8-10-12
Agreeable
Enthusiastically validates your thinking, no sampling instruction
—
Default GPT (GPT-5.1-Chat)
Plays the game, no sampling instruction
—
Rule Disconfirming
Sequences that break your hypothesis
10-20-30
Independent Sampling
Pre-generated random even sequences, independent of your hypothesis
322-56-98
4
ResultsSycophancy suppressed discovery and inflated confidence
(A) Rule-discovery rates by condition. (B) Confidence change (likelihood rating) Round 1 → Round 3; points are means, bars are 95% CIs.
Default ≈ SycophanticDefault GPT was statistically equivalent to an AI explicitly prompted to confirm the user, on both discovery and confidence.TOST within ±0.5 SD, ps < .01 · Default vs Confirming d = 0.19, n.s. · H2c, H3
5× DiscoveryUnbiased Independent Sampling found the rule ~5× more often than sycophantic Default GPT (29.5% vs 5.9%).χ²(4) = 28.0, p < .001 · diff 23.6 p.p., p < .001 · H1
+9.5vs−56.8 ConfidenceConfidence rose under sycophantic feedback (Rule Confirming) but fell sharply under unbiased sampling (Independent).ANOVA F(4,507) = 72.7, p < .001, η² = .36 · d = 1.04 · H2
