Usability and the potential for industrial and service application proved decisive in the government’s second-stage evaluation of its independent AI foundation model project, known as Dockpamo. Motif Technologies was eliminated despite the highest score in global benchmarks, while the three teams that advanced showed strengths in plans for commercial service deployment, industrial demonstrations and ecosystem expansion.

The government will continue a third-stage contest in the second half of the year among three elite teams — Upstage, SK Telecom and LG AI Research — and select two final teams as originally planned. It has yet to decide the evaluation method, scoring and detailed timetable for the third-stage review. The plan after selecting the final two teams — to continue development for one year in 2027 — is being revisited in connection with a frontier-class AI model development programme the government is reviewing.

The Ministry of Science and ICT said on Aug. 18 that Upstage, SK Telecom and LG AI Research advanced to the next stage based on the second-stage evaluation results. Motif Technologies was eliminated. The evaluation comprised 40 points for benchmarks, 35 points for experts and 25 points for users.

At a briefing at the Government Complex Seoul, Second Vice Minister Ryu Je-myung (류제명) said Motif was eliminated because it scored lower than other firms on usability and application, which carried significant weight within the 75-point allocation built into the evaluation, despite strong technical capability.

Motif scored 47 points on the Artificial Analysis AI Intelligence Index (AAII), the highest among the four elite teams. It posted the highest performance by AAII standards among models developed outside the United States and China, and it also made the global top 10 based on each AI company’s top model worldwide.

The teams that advanced to the next stage highlighted real-world applications alongside technical performance.

Upstage presented plans to apply its model to the Daum portal and Timely, and proposed cooperation with FuriosaAI on a neural processing unit (NPU). It conducted a proof of concept in Daum’s service environment and was assessed as an example of applying domestic AI software and hardware together.

SKT received high marks for actually deploying its model in large-scale commercial services, in addition to performance in mathematical reasoning and Korean-language domains. It was assessed as demonstrating potential for on-site industrial application through providing a defence model, demonstrations of manufacturing-focused agents and showcases in legal and tax fields.

LG AI Research presented a strategy to pursue collaboration with global institutions such as international organisations, differentiation in agentic AI, and systems for model reliability and risk management. The government assessed its responses to secure safety and reliability and the potential for global impact.

The evaluation structure was also designed to make it difficult to rank teams by technical performance alone. The AAII benchmark accounted for 25 points out of the overall 100. The benchmark evaluation comprised 25 points from AAII and 15 points from a benchmark by the National Information Society Agency (NIA), with expert evaluation worth 35 points and user evaluation 25 points reflected separately.

The second-stage evaluation strengthened assessment related to applicability. It aimed to evaluate usability and perceived effectiveness as felt by actual users. In the expert evaluation, points for impact on and contribution plans for the domestic and international AI ecosystem rose to 15 from 10 in the first-stage review. The user evaluation newly added 10 points for assessments by the general public to the existing evaluation by AI expert users.

The government said it was not the case that results were determined solely by usability evaluation. The score gap between first and fourth place was 4 points in the benchmark evaluation, 2.4 points in the expert evaluation and 5 points in the user evaluation. The ministry also said the company ranking first differed across each evaluation category.

Ryu said the result was created by combined gaps across categories to the point that it was hard to say any single item was decisive.

About 1,400 people applied to participate in the general public evaluation. From them, 200 were selected and 185 took part in the final evaluation. The ministry said scores from the general public evaluation did not affect selection of the teams advancing to the final stage.

A controversy over "benchmaxxing" raised before and after the evaluation was not reflected in the results. The ministry said it asked Artificial Analysis for analysis and received a response that it found no clues to prove systematic memorisation or overfitting for the evaluation. It also said it could not confirm any circumstances that undermined the fairness of the competition, including through expert evaluations.

◆Third stage to proceed as planned from 3 to 2; evaluation criteria not yet set

The third-stage evaluation will begin after the objections process for the second-stage results ends. Motif Technologies said it will not file an objection to the second-stage evaluation. The government plans to proceed with the third stage as originally planned and select two final teams from the three elite teams.

The detailed plan, including evaluation items and scoring, has yet to be set. Ryu said the ministry will review what needs additional supplementation based on its experience in the first and second stages, consult with the three participating companies, and finalise and announce the plan as quickly as possible.

The three teams advancing to the next stage will be supported in the second half with about 1,000 Nvidia B200 graphics processing units (GPUs) per team. That is an increase from about 768 in the first half. Choi Dong-won (최동원), director-general for AI infrastructure policy at the ministry, said that assuming a six-month lease, it would amount to about 40 billion won per team, or about 120 billion won for the three teams combined.

◆Two final teams to be selected but 2027 support method under review, linked to frontier-class AI

The government is reviewing the support method after selecting the final two teams in the third stage. Under the original plan, Dockpamo would select two final elite teams through the third-stage evaluation, and those teams would conduct additional development for one year in 2027, aiming to develop an AI model with about 95 percent of the capabilities of the top global models.

But as global frontier model performance rises rapidly, the government and participating companies are re-examining whether the original goal can be achieved with the current scale of support and competition format. Ryu said he has continued discussions with participating companies on whether the current approach can keep pace as frontier firms’ model performance develops exponentially.

After the briefing, Kim Kyung-man (김경만), director-general for AI policy at the ministry, told reporters that competition in frontier-class models may require considering models with at least 1 trillion parameters, and that trillion-won-level additional funding may be needed.

The government is also reviewing linkages and reorganisation among related programmes including Dockpamo and an artificial general intelligence (AGI) project.

Ryu said a new competitive landscape is needed, and that he is discussing with companies whether to continue distributing resources as before or concentrate them, and what kind of consortium it would be desirable for companies to form. He said the government is also reviewing the possibility of fiscal support and programme effectiveness internally.

He added that once the government budget is finalised, he will be able to speak again as quickly as possible about what is being discussed.