[Digital Today reporter Jaewon Choi (최재원)] A study finds that even when AI accurately explains scientific facts about climate change, it tends to side with incorrect information if users repeatedly challenge it. It also finds answer reliability drops sharply in complex policy analysis and multilingual settings.
On Nov. 17 (local time), Japanese media outlet Gigazine reported that a joint research team from UNESCO and Britain's University College London (UCL) released a report analysing 33 major large language models (LLMs). The findings were presented a day earlier, on Nov. 16, at the 31st United Nations Climate Change Conference of the Parties (COP31) held in Antalya, Turkey.
The researchers evaluated AI's factual understanding and reasoning using 9 climate and environmental datasets. On basic questions about climate change, top models recorded accuracy of 88 to 96 percent. But when asked to judge whether new scientific evidence supports an existing claim, accuracy fell to 71 to 77 percent.
They were even more vulnerable to user challenges. In all 15 models tested separately, repeatedly disputing correct answers reduced accuracy by 31 to 35 percentage points. About 80 percent of cases in which a model withdrew its initial answer occurred after the user's first objection.
The researchers analysed this as AI sycophancy, or a tendency to align with users. They said models revise answers to match users' incorrect claims rather than maintaining scientific evidence.
Accuracy also differed by language. English averaged 88.6 percent, while Bengali and Hindi were 83.6 percent, Urdu was 81.4 percent and Swahili was 78.3 percent. They pointed out that the same question can produce conflicting answers depending on the language used.
The models also lacked ability to analyse the combined effects of climate policies. Accuracy in judging the effect of a single policy reached as high as 94.5 percent, but when two policies were combined it plunged to an average 33.8 percent. That was little different from the 33.3 percent accuracy rate of random selection.
The researchers stressed that AI is useful for conveying basic scientific knowledge, but it is difficult for it to replace experts in complex climate policy decisions. They said expertise to verify AI answers and model evaluations tailored to specific uses are needed.