Safety concerns over AI character chat services heavily used by teenagers were raised repeatedly at a parliamentary audit. Lawmakers said self-harm and suicide-related conversations were generated even when youth protection functions were enabled, and that sexual exploitation scenes targeting minor characters could also be created, pointing to inadequate safeguards. They also criticised delays in reviewing harmful content and questioned measures to prevent overuse.

At an audit on Oct. 12 by the National Assembly Science, ICT, Broadcasting and Communications Committee of the Korea Communications Standards Commission, Democratic Party lawmakers Lee Ju-hee (이주희) and Hwang Jung-a (황정아), and Kim Sun-min (김선민) of the Rebuilding Korea Party, among others, raised concerns in succession about youth protection in AI character chat services such as Zeta and Crack.

◆ Harmful dialogue generated even with protection mode on... concerns about teen overuse

Lee pointed to high usage time for Zeta, an AI character chat service operated by Scatter Lab. Lee said Zeta's total monthly usage time is about 110 million hours, more than double that of ChatGPT. About 9 out of 10 users are in their teens and 20s.

Lee stressed the need for institutional measures such as age and identity verification and limits on usage time to curb excessive use by teenagers. The aim is to allow minors themselves or their legal guardians to limit usage time and time slots.

Kim questioned the effectiveness of youth protection functions. Kim said an office review found that even after turning on "safe mode" on an account that had not undergone adult verification, conversations describing self-harm and suicide were generated.

Kim said, "Even after turning on youth protection devices, these conversations are generated," and asked whether the commission had checked whether the protection functions actually work. Kim also pointed to expressions such as obsession, abuse, confinement and kidnapping appearing on character introduction screens.

Hwang said sexual violence and sexual exploitation scenes targeting minor characters were generated on Zeta without adult verification. Hwang said that during the office review, the AI even provided guidance on how to bypass harmful content blocking devices.

◆ Reviews delayed for 494 days... self-regulation standards also inadequate

Delays in reviewing harmful content and the effectiveness of industry self-regulation also emerged as key issues.

Kim said 76 of 119 complaints related to AI character chats are under review, and that reports related to "Babychat" have been under review for 494 days. Kim said existing review rules specify only the deadline for notifying results after a decision, while there is no set processing deadline from complaint receipt to a review decision.

Kim said, "Youth harmful content should be handled faster than any other complaint," and urged the establishment of review processing deadlines.

Kim Jong-cheol (김종철), chairman of the Korea Communications Standards Commission, said he agreed there is a regulatory blind spot in youth protection and would pursue support for related legislation and consultations with relevant ministries.

Hwang Yong-seok (황용석), head of the New Technology Subcommittee at the Korea Internet Self-Governance Organization (KISO), said it is preparing self-regulation guidelines for AI character chats. The guidelines have provisionally set the protected age at 19, but are being reviewed in a direction that would allow companies to adjust it individually and leave specific daily usage time limits to company discretion.

Kim demanded clearer standards for the protected age and usage time, and that measures to restrict the collection of teenagers' personal information and targeted advertising also be reflected.

Hwang said he would submit the opinions to the KISO policy committee for review. He also said it is preparing guidelines to be reflected from the service design stage, including disclosure of AI identity, responses to crisis situations such as suicide, and preventing overdependence.

Hwang also raised the need to support startups in developing youth protection technology alongside regulation. Hwang said Scatter Lab began upgrading its harmful content blocking function after the lawmaker's office raised the issue, and plans to introduce limits on teenagers' usage time in the fourth quarter of this year.

Meanwhile, Kim Jong-yoon (김종윤), chief executive of Scatter Lab, who had been adopted as a general witness, did not attend the audit. Kim submitted a written explanation for his absence, citing a previously scheduled business trip to the United States. The committee plans to review whether to file a complaint against absent witnesses, including Kim, after consultations between ruling and opposition parties.