A study found that users who look up information with a generative AI chatbot may have a lower learning effect than users of traditional search engines.

On June 15 local time, Japanese outlet ITmedia reported that researchers affiliated with institutions including Georgia Tech and the University of Michigan released results from an eight-day experiment comparing learning effects of ChatGPT and Google Search in a paper titled "Learning by Chatting? Investigating the Impact of Generative AI on Information Seeking and Learning".

The researchers focused on a shift in information seeking from keyword search to conversational questions as generative AI spreads. They divided participants into 2 groups and compared learning processes on the topic of "nutrition and diet planning". There were 80 participants in total, and 35 were included in the final analysis. One group used ChatGPT and the other used conventional Google Search. The AI summary feature was disabled in Google Search.

The results were unfavorable to the ChatGPT group. The researchers said the group using ChatGPT tended to show overall lower learning effects than the Google Search group. Differences were particularly seen in the ability to apply learned knowledge to other situations and the ability to compare and evaluate multiple pieces of information.

The researchers presented 2 factors behind the results. The first was how generative AI answers. ChatGPT tended to present a finished output before basic principles or operating structure. Using the example of finding a balanced diet, the researchers explained that Google Search made it easier to reach materials that explain nutrient ratios and their rationale, while ChatGPT often immediately produced specific meal plans or shopping lists. They said the learning step of understanding "why" could be skipped in this process.

The second was a change in information-seeking behavior. The researchers said a conversational interface can narrow a user's view and reduce the process of reviewing information. ChatGPT users tended to start with personalized, narrow questions such as "What diet is right for me?" By contrast, a search engine encouraged users to broaden perspectives by moving across multiple websites and to check information reliability directly.

The researchers noted that because AI gives a "single neatly summarized answer," users make less effort to compare multiple sources or seek other views. They said it also raises the possibility of more passive learning in which users accept the provided answer as is. The study concluded that the ChatGPT group's weakness was pronounced in the ability to compare and evaluate multiple pieces of information.

Another notable point was that convenience did not translate into learning satisfaction. Participants who used ChatGPT appeared to feel greater dissatisfaction during the learning process. They had to keep changing how they asked questions to get the desired level and format of information, and they felt it was difficult to learn by filtering information at their own pace. The researchers said learning fatigue could increase when users do not have sufficient control over information, contrary to expectations that AI would make things easier by replacing search.

The study shows that while generative AI can increase the speed of access to information, it does not automatically raise the quality of learning. It found that in learning situations where understanding principles, verifying information and comparative judgment are important, search engine-based exploration may still have strengths. It suggested that when using generative AI as a learning tool, users may need not only to get a finished answer immediately but also to check sources broadly and examine the grounds.