Kakao said an artificial intelligence (AI) evaluation metric it developed in-house has had its technological capabilities recognised by an international conference.

On Monday, Kakao said its in-house orchestration benchmark has been accepted as a paper at ICLR 2026, one of the world’s three major AI conferences. ICLR is a leading academic conference watched by global big tech companies. About 19,000 papers were submitted this year, and competition was intense, with only the top 28 percent accepted.

Kakao said it developed the benchmark by focusing on the trend of large language models (LLMs) evolving into agentic AI. Agentic AI goes beyond simple question-and-answer functions to plan and coordinate tasks requested by users.

The study focuses on assessing multi-step and multi-domain processing capabilities in real service environments. Kakao said existing benchmarks were limited to fragmentary performance evaluations, making it difficult to verify the ability to interpret complex requests and use tools. It presented a new framework that precisely evaluates performance by separating task planning from tool execution.

The benchmark includes 17 service domains, such as travel, shopping, finance and schedules, and more than 100 virtual tools. It reflects real conversational flows, such as changes to user requests or follow-up questions, to improve practicality.

The evaluation data supports two languages, Korean and English. All data were manually verified, and Korean in particular reflects cultural characteristics and context to secure accuracy and reliability. Kakao released the paper as open source on GitHub to help energise the AI ecosystem.

A Kakao official said, "The orchestration benchmark is a key tool for evaluating whether agentic AI operates stably in real service environments, and it is a meaningful research outcome at this point as agentic AI evolves to judge and carry out tasks on its own." The official added, "We plan to continue research to strengthen collaboration among agents and orchestration capabilities, and we will push to advance technologies that align with Kakao’s direction for agentic AI."