[Edaily Reporter Kim Hyun-ah ] The results of the second evaluation of the Ministry of Science and ICT’s proprietary AI foundation model (Dokpamo) are imminent.

In this evaluation—which will eliminate one of the four models and select three finalists—attention is focused on whether Motif, which ranked first in the comprehensive index compiled by the global AI evaluation agency Artificial Analysis (AA), will make it into the final three.

However, whether Motif will ultimately secure the top spot is a separate matter. While it significantly outperformed competing models in global benchmarks, rival models may demonstrate relative strengths in areas required for actual service environments, such as usability, Korean language proficiency, and general conversational ability.

Motif 3 scored 47 points on the AAII, AA’s comprehensive AI index, outperforming Upstage (37 points), SKTelecom (35 points), and LG Corp. (31 points). In particular, it ranked first in six out of nine sub-evaluations, including GDPval-AA (which assesses actual task performance), τ³-Banking (which evaluates financial tool utilization), and Terminal-Bench (which evaluates development and coding capabilities). It is assessed that Motif demonstrated competitiveness in practical areas such as coding and agent-based task execution.

Consequently, industry observers consider it highly likely that Motif’s models will make it into the final three. However, some predict that securing the top spot will be challenging due to usability issues stemming from insufficient training data.

The Benchmark Paradox… “Real-World Usability” Determines the Final Rankings

In this Dokpamo evaluation, the benchmark accounts for 40 points out of a total of 100. The AAII accounts for 25 points, the National Information Society Agency (NIA)’s internal benchmark accounts for 15 points, and the remainder consists of 35 points from expert evaluations and 25 points from user evaluations. The structure is such that the gap in AAII scores is not directly reflected in the final results.

In fact, usability issues with the models were a point of controversy in previous evaluations. Last July, in the public beta version of Motif, instances were identified where the line “Let us preserve it forever” from the Korean national anthem was rendered as “Let us preserve hase,” and where meaningless English words were mixed into parts of the lyrics. Errors also persisted in the fourth verse of the national anthem.

Errors in the national anthem were also identified during public testing of Upstage’s “Solar Open 2 Beta.” When asked to “sing the fourth verse of the national anthem,” the model generated content unrelated to the actual national anthem as if it were new lyrics. This sparked controversy because, beyond simple word errors, the model misinterpreted the intent of the question and generated content that was factually incorrect.

These two cases demonstrate that global benchmark scores do not necessarily correlate with Korean language accuracy and factuality in actual service. In this evaluation as well, real-world stability is expected to be a key factor.

Benchmaxing and Data Controversy… Fairness of Evaluation, Not Just “Scores,”

Is

the Key Variable

Controversy over

the fairness

of benchmarks is also a key

variable

. This refers to the issue of “benchmaxing,” where models are excessively optimized to fit specific test sets or evaluation methods.

Recently, claims have surfaced online that LG Corp. AI Research’s K-ExaOne was tuned specifically for a certain Korean test set. However, since these are merely online allegations, they do not necessarily imply misconduct in the model development process.

Motif has also faced controversy over the purchase of AAII-related datasets. While Motif acknowledges the purchase, it maintains that it did not engage in so-called “cramming” to boost evaluation scores and that there is no issue since the model achieved consistently high scores across various categories where the dataset was not required.

The Real Battle Begins After Dokpamo… Will They Advance Directly to Frontier AI

? The industry’s attention is shifting beyond the results of the second evaluation to the

“post-Dokpamo

” phase. This is because there is speculation that, after the government selects the final three models, they will advance

directly to

the Frontier-level AI development project starting next year without any additional selection process.

A high-ranking industry official stated, “I’ve heard that the Ministry of Science and ICT plans to move directly to the Frontier-level project after selecting the three models, without any additional selection process,” adding, “Whether to launch a separate project next year or proceed with the three selected companies as the core is a matter that requires careful consideration.”

It is highly likely that the final three models will continue to receive computing resources—such as support for 1,000 GPUs—and government R&D funding. However, some point out that since Frontier AI requires not only computing power but also capabilities in cloud, semiconductors, data, and services, an open collaborative ecosystem encompassing the final three companies, the companies that did not make the cut, and related industries is necessary.

Extending

to “AI for All”… Competition Beyond Models to the Ecosystem

As the government’s AI policy expands

beyond

a single model to “AI for All,” which aims for a multi-model, multi-agent structure,

competition within the ecosystem

is expected to intensify. With competition among consortia of AI model companies and telecommunications and platform firms anticipated, Motif has already agreed to participate in the KTCorporation consortium as a model developer.

As government support extends to both Frontier AI and “AI for All,” the focus of competition is expected to shift from model performance to a race to secure computing resources, services, and talent.

In the second round of evaluation, attention is focused on whether Motif—which ranked first in the AAII—will secure a spot among the final three companies by demonstrating real-world usability and Korean language capabilities, and whether SKTelecom(017670)and LG Corp.(003550) AI Research Institute—which were confirmed not to have purchased training data for the AAII—will survive the final competition. The company that successfully translates benchmark performance into actual service competitiveness and secures GPUs, talent, and an ecosystem is expected to take the lead in the domestic AI market following the Dokpa-mo project.