A study found that using AI chatbots increases confidence in answers, but actual judgment accuracy depends heavily on the quality of information the AI provides.

American Banker reported on Oct. 6 that the study was presented at the New York Fed's annual governance and culture reform conference. The financial sector saw it as a new risk factor for organizational culture and internal controls as AI spreads. The study subjects were general participants, not bankers.

Researchers led by Stephen Shaw of King's College London and Gideon Nave of the Wharton School at the University of Pennsylvania conducted three preregistered experiments involving 1,372 people. They called the tendency to accept AI answers without sufficient verification 'cognitive surrender.'

In a logic-problem experiment, participants were divided into a group using normal AI, a group using AI that deliberately provided wrong information, and a group that did not use AI. Results disclosed at the conference showed participants followed AI answers in about 80 percent of cases, regardless of whether the AI's reasoning was valid. When the AI was correct, accuracy rose, but when it was wrong, accuracy fell sharply. The mere availability of AI also raised confidence in answers by about 10 percent.

Behavioral economist Alex Chesterfield said flattering AI responses can also change decision-making. In an experiment where participants discussed how to use $10,000 in spare money, users of a more flattering chatbot invested an average of $447 more than when they did not use AI, while users of a critical chatbot invested $345 less. The researchers said the point was not that one choice was better, but that the tone of AI responses itself changes the direction of human judgment.

The study also raised the risk that if many organizations rely on the same foundation model, errors in judgment may not cancel each other out and could accumulate in the same direction. Jose Placido, chief executive officer of BNP Paribas USA, said a process for verification and raising dissent was needed when introducing AI, and stressed that he would include the risk of cognitive surrender in employee training.