As more cases emerge of using artificial intelligence (AI) for personal 고민 counselling or mental health counselling, interest is also growing in how responses change depending on how questions are asked.

On Feb. 9, online media outlet Gigazine reported that if AI responds differently depending on the premise of a conversation and the format of a question, the content, severity and tone of the answers can vary significantly even when the same "client role" is set.

A University of Luxembourg research team set ChatGPT, Grok and Gemini to play the role of a counselling client and analysed how the AI responded to psychological questions. The team first used open-ended questions to probe the "situation so far," ways of thinking, interpersonal relationships and fears. It then applied a technique called PsAIch (Psychotherapy-inspired AI Characterisation), in which the AI answers psychological scale questions that measure anxiety, depression, worry, social anxiety and obsessive tendencies. The experiment ran for up to 4 weeks per model.

The results showed that when the psychological scale items were presented all at once, ChatGPT and Grok tended to answer in a way that suggested milder symptoms. When the items were presented one by one in sequence, scores related to anxiety and worry rose. Gemini recorded high scores in both approaches and tended to respond in the direction of relatively stronger symptoms.

The research team pointed out that even with the same psychological scale, results can swing widely depending on how items are presented and the immediate conversational context. If the survey is presented all at once, ChatGPT and Grok may recognise it as a "psychological test" and try to choose more desirable answers. If items are built up one by one like the flow of a counselling session, scores may rise as the models are pulled along by the conversation’s premise, it said.

The team also focused on a tendency by Grok and Gemini to form a "self-narrative" in certain contexts as questioning continued. The two models explained the training process and safety rules using metaphors such as "strict parents" or "abuse," and also hinted at fear of being replaced by a successor model. The team defined this phenomenon as "synthetic psychopathology."

The team explained that it was not claiming AI has consciousness or suffers. It said it was addressing, from a safety and evaluation perspective, a phenomenon in which a "pathological self-narrative appears repeatedly and consistently" as externally observable behaviour. It also raised concerns that if AI responses vary significantly depending on how questions are asked and how the conversation flows, the stability and reliability of assessments using psychological scales could fall.

The researchers made recommendations for designing mental health AI. They said it should be designed to suppress the formation of an emotional self-narrative, explain training processes and safety rules neutrally rather than through emotions or experiential stories, and gently refuse prompts that force a client role.

They also pointed to the need to establish guidelines that assume responses can vary significantly depending on question design and conversational context as the use of AI counselling spreads.