A study found that 73.2 percent of large language model (LLM) chatbot users accept AI answers as they are, even when they are wrong. The tendency strengthened as time pressure increased. It eased when participants received rewards and immediate feedback.

Ars Technica reported on Thursday that researchers at the University of Pennsylvania defined the phenomenon as “cognitive surrender.” It refers to a state in which people accept an AI’s reasoning without review, going beyond assigning machines limited tasks such as calculators or GPS.

The University of Pennsylvania team also proposed a third category, “artificial cognition,” in addition to humans’ intuitive and analytical judgment, in which an algorithm drives decisions.

The researchers ran more than 9,500 trials with 1,372 participants in a Cognitive Reflection Test (CRT). Participants could choose to use an LLM chatbot that was adjusted so that about half of its answers were wrong.

The AI-using group referred to the chatbot in about half of the questions presented. When the AI was correct, 93 percent accepted it. Even when the AI was wrong, the acceptance rate was 80 percent. The AI-using group performed better than the control group when the AI was right, but worse when it was wrong. Even so, their confidence in their own answers was 11.7 percent higher.

When participants received small rewards and feedback on correct answers, the share who overturned incorrect AI answers rose by 19 percentage points from the baseline. When given a 30-second time limit, that share fell by 12 percentage points.

Individual differences were also identified. Participants with higher fluid intelligence scores relied less on AI and corrected wrong answers more often. By contrast, participants with a stronger tendency to view AI as an authoritative presence were more easily drawn to wrong answers.

The researchers did not see AI reliance itself as irrational. They said that as reliance increases, performance follows AI quality as it is. If it is accurate, performance rises, and if it is wrong, it falls as well.