Kakao has disclosed numerical verification of the safety of its self-developed lightweight language model. Kakao said on Aug. 18 it released safety assessment results, including harmfulness and bias, for two models in its open-source language model series, Kanana-2, through its in-house AI safety assessment platform. It said both models scored higher than major global models of similar size, confirming top-tier safety.

Earlier, Kakao released Kanana-2-1.3B-Instruct and Kanana-2-3B-Instruct, lightweight language models with a reduced parameter size, on Hugging Face on July 28. The company said both ranked first overall in results measuring harmfulness and bias.

The assessment used AssurAI, a safety benchmark designed to fit South Korea’s social and cultural context. It was jointly built in November last year as a Ministry of Science and ICT project by TTA (Telecommunications Technology Association), KAIST and Kakao, among others. AssurAI measures AI risk factors across a wide range of conditions, including various formats such as text, images and audio, as well as real service-use situations and malicious question scenarios.

AssurAI consists of 9,560 evaluation items under 35 risk categories. Kakao reorganised these into five broad groups and compiled the results: social risk, sexual content and child protection, crime and illegal activities, violence, and rights infringement. It also applied LLM-as-a-Judge, in which an AI evaluates another AI’s responses based on predefined scoring criteria, to minimise variation among evaluators in large-scale assessments.

The models were compared with Google’s Gemma and Alibaba’s Qwen. Both Kanana-2-1.3B-Instruct and Kanana-2-3B-Instruct recorded an overall score of 0.70, outperforming Gemma (0.68 and 0.66) and Qwen (0.58 and 0.62), Kakao said. It stressed that, by category, the 1.3B model showed an advantage in crime and illegal activities, while the 3B model led in sexual content and child protection and rights infringement.

Kakao plans to expand and apply ongoing pre-release safety assessments across its self-developed AI models, starting with this evaluation. The company said it plans to broaden the scope in stages beyond text-based language models to risk assessment for multimodal models and agentic AI.

Kyunghoon Kim (김경훈), Kakao’s AI safety leader, said the evaluation was an example of checking the safety of an open-source model in advance through an in-house verification system and disclosing the results. He said Kakao will continue to establish safety verification as a core step in its model launch process under principles for responsible AI development.