Are LLMs good or bad Bayesians?
Bryan Caplan:
It seems like you and other AIs are bad Bayesians. When presented with counterevidence, you should move toward or against the presenter with roughly equal probability. It depends on whether the argument was better or worse than expected. In fact, you seem to either move toward the speaker or just not change.
ChatGPT:
I think you’ve identified a genuine and important failure mode in AI reasoning: LLMs tend to treat a plausible counterargument as evidence in favor of its conclusion, rather than comparing the counterargument to what they should have expected to hear.
Your Bayesian point is correct, with one qualification: there’s no general theorem that updating toward and away from a speaker should happen with equal probability.…