Naver Cloud and NC AI's elite team were eliminated from the "independent AI foundation model" project to pick a national AI model. Naver Cloud cleared the first evaluation on benchmark scores but failed to meet the independence criteria.

The Ministry of Science and ICT, the National IT Industry Promotion Agency (NIPA) and the Telecommunications Technology Association (TTA) on Wednesday released the first-stage evaluation results for the independent AI foundation model project.

Five consortiums joined the project: Naver Cloud, Upstage, SK Telecom, NC AI and LG AI Research. The ministry, NIPA and the five elite teams held multiple in-depth discussions to set the first-stage evaluation method and criteria.

LG AI Research tops all three benchmark evaluations

The first-stage evaluation conducted benchmark, expert and user assessments. It comprehensively evaluated AI model performance (AI Frontier Index) and usability and spillover impact (AI Diffusion Index), including real-world applicability, cost efficiency such as model size, and plans for impact on AI ecosystems at home and abroad.

The 40-point benchmark assessment consisted of the NIA benchmark (10 points), the global common benchmark (20 points) and the global individual benchmark (10 points).

The NIA benchmark evaluated math, knowledge and long-context understanding, and also assessed reliability and safety in collaboration with the AI Safety Institute. The global common benchmark selected 13 globally recognised benchmarks across areas including agents, math, knowledge and reasoning, and instruction following. The global individual benchmark assessed five benchmarks comparable with each elite team's global target model, at a state-of-the-art level.

In the NIA benchmark, SK Telecom and LG AI Research each scored 9.2 out of 10, the highest among the five elite teams. In the global common benchmark, LG AI Research scored 14.4 out of 20, the highest. In the global individual benchmark, Upstage and LG AI Research tied with 10.0 out of 10, the top score.

The 35-point expert assessment was conducted by an evaluation committee of 10 external AI experts from industry, academia and research, which carried out a long-term in-depth review of materials submitted by each team. It evaluated development strategy and technology, development results and plans, and spillover impact and contribution plans.

The committee analysed technical reports disclosed by the five elite teams for each model, as well as AI model training status log files, to evaluate technical capabilities including development processes and independence. LG AI Research scored the highest with 31.6 out of 35. The participating teams averaged 28.56 points.

For the 25-point user assessment, 49 AI expert users, including AI startup CEOs, took part. They analysed real-world applicability and inference cost efficiency based on AI-use websites built with the elite teams' AI models. LG AI Research scored the full 25 points, more than 4 points above the average of 20.76.

The ministry said a consolidation of the scores placed LG AI Research, Naver Cloud, SK Telecom and Upstage among four teams, excluding NC AI.

Naver's elimination surprises as independence analysis falls short

The ministry eliminated two teams in the first-stage evaluation, deviating from its initial plan to drop one elite team. Naver Cloud was eliminated along with NC AI, which received a low benchmark score. Naver Cloud cleared the benchmark threshold but failed to meet the government's independence criteria, leading to its elimination in the first evaluation.

The ministry has defined an independent AI foundation model in the project notice as "a domestically developed model that carries out everything from model design to pre-training, rather than a derivative model developed through overseas model fine-tuning."

From a technical perspective, the ministry aims for an independent implementation of AI models that performs end-to-end training through processes including original AI model architecture design, securing and processing large volumes of data on its own, and applying proprietary learning algorithm techniques.

A ministry official explained, "Using open source is a general trend in the global AI ecosystem, but the basic condition for an independent AI model accepted across the domestic and international AI industry and academia is to develop the AI model by training after initialising the weights."

Accordingly, the evaluation set as a minimum condition for securing model independence that even if an elite team strategically uses verified open source, it must train and develop after initialising the weights.

From a policy perspective, it aimed to ensure capabilities to develop and advance AI models on its own at any time and to control operation and use of AI models independently in any situation, in order to resolve the possibility of concerns such as leaks of state secrets or threats to national security.

That is, it must be able to develop an AI model entirely with Korean technology, or use open source without licence restrictions to develop and advance it on its own. It must also be free from external control or interference arising from the use of open source.

From an ethical perspective, it stressed compliance with licence policies, including disclosing references for the developed AI model. The ministry said there is a need to "promote sound development by securing trust in the AI ecosystem, strengthening open verification and improving transparency."

After considering these three perspectives, the Naver Cloud elite team's AI model was judged not to meet the independence criteria. Expert evaluators also raised issues about limitations in independence. The ministry said it decided, taking these factors into account, that the Naver Cloud elite team did not meet the conditions for an independent AI foundation model.

As a result, after the first-stage evaluation, LG AI Research, Upstage and the SKT elite team advanced to the second stage from the original five elite teams.

Ministry to open additional call for one more elite team

The ministry will select one additional elite team, opening the opportunity to consortiums that applied in the first call for the independent AI foundation model, consortiums not included among elite teams after the first-stage evaluation (the Naver Cloud and NC AI consortiums), and other capable companies. Through this, it aims to secure a four-team competition structure in the first half of this year.

The one additional elite team will receive GPU and data support and be granted the name "K-AI company," among other opportunities to challenge the development of an independent AI foundation model. The ministry will move quickly on administrative procedures and push ahead with an additional call for one elite team.

The ministry said, "This project is a historic challenge for South Korea to stand up to global AI competition with independent technology," adding it will "concentrate all available national capabilities and resources so that we build a sustainable and healthy AI ecosystem and South Korea can stand at the forefront of global AI technology competition."