KT has introduced a benchmark that reflects social issues and cultural context in South Korea to assess the safety of multimodal artificial intelligence models.

KT on June 16 released KSAFE-MM, a safety benchmark for multimodal large language models developed jointly with Korea University.

KSAFE-MM is a dataset designed to assess the safety of multimodal AI models that integrate and process various data such as text, images and audio, tailored to the Korean language and Korean cultural environment. It consists of KSAFE-MM-G, which converts global common risk factors into the Korean cultural context, and KSAFE-MM-C, which reflects issues unique to South Korean society such as rental deposit fraud and the Dokdo dispute.

The total number of evaluation samples is 14,135. Researchers from KT and Korea University validated safety across 12 global multimodal large language models, including Gemma and HyperCLOVA X.

KT also built a general-purpose pipeline that automates the process from data collection to the creation of evaluation items. It addressed limitations of existing safety benchmarks, which are centered on manual review and require substantial cost and time. The pipeline implements a four-step automated process covering the full workflow, from collecting sensitive topics based on local communities to template-based query generation, synthetic image generation and jailbreak query generation.

This allows safety benchmarks that reflect local characteristics to be built quickly without experts from a specific cultural sphere. Researchers also confirmed the potential to expand to other cultural settings through a pilot experiment, JSAFE-MM-C, applying the same pipeline to Japanese.

The research results and the benchmark were released on the paper-sharing platform arXiv and the AI open-source platform Hugging Face.

Jae-hyung Park (박재형), head of the Frontier AI Lab at KT's AX Future Technology Institute, said releasing the safety benchmark helps build a foundation for the AI safety research ecosystem to develop together. He said he hopes KSAFE-MM becomes a common standard in academia and industry for verifying AI safety in the Korean language and Korean culture.