[Edaily Reporter Kim Hyun-ah ] The results of the second evaluation of the Ministry of Science and ICT’s proprietary AI foundation model (Dokpamo) are imminent.

In this evaluation—which will eliminate one of the four models and select three finalists—attention is focused on whether Motif, which ranked first in the comprehensive index compiled by the global AI evaluation agency Artificial Analysis (AA), will make it into the final three.

However, whether Motif will ultimately take first place is a separate matter. While it significantly outperformed competing models in global benchmarks, rival models may demonstrate relative strengths in areas required for real-world service environments, such as usability, Korean language proficiency, and general conversational ability.

Motif 3 scored 47 points on AA’s comprehensive AI index, AAII, outperforming Upstage (37 points), SKTelecom (35 points), and LG Corp. (31 points). Notably, it ranked first in six out of nine sub-categories, including GDPval-AA (which evaluates actual task performance), τ³-Banking (which assesses financial tool utilization), and Terminal-Bench (which evaluates development and coding capabilities). It is assessed that Motif demonstrated competitiveness in practical areas such as coding and agent-based task execution.

Consequently, industry observers consider it highly likely that Motif will make it into the final three models. However, some predict that securing the top spot will be challenging, as usability issues stemming from insufficient training data could act as a wild card.

◇The Paradox of Benchmarks… “Real-World Usability” Determines the Final Rankings

In this Dokpamo evaluation, the benchmark accounts for 40 points out of a total of 100. The AAII benchmark is worth 25 points, the National Information Society Agency (NIA)’s internal benchmark is worth 15 points, and the remainder consists of 35 points for expert evaluation and 25 points for user evaluation. The structure is such that the gap in AAII scores is not directly reflected in the final results.

In fact, usability issues with the models were a point of controversy in previous evaluations. Last July, in the public beta version of Motif, instances were identified where the line “Let us preserve it forever” from the Korean national anthem was rendered as “Let us preserve hase,” and where meaningless English words were mixed into parts of the lyrics. Errors also persisted in the fourth verse of the national anthem.

Errors in the national anthem were also identified during public testing of Upstage’s “Solar Open 2 Beta.” When asked to “sing the fourth verse of the national anthem,” the model generated content unrelated to the actual national anthem as if it were new lyrics. This sparked controversy because, beyond simple word errors, the model misinterpreted the intent of the question and generated content that was factually incorrect.

These two cases demonstrate that global benchmark scores do not necessarily correlate with Korean language accuracy and factuality in actual service. In this evaluation as well, real-world stability is expected to be a key factor.

◇ Benchmarking and Data Controversies… Fairness of Evaluation, Not Just “Scores,” Is the Key Variable

Controversy over the fairness of benchmarks is another variable. This refers to the issue of so-called “benchmaxing,” where models are excessively optimized to fit specific test sets or evaluation methods.

Recently, claims have surfaced online that LG Corp. AI Research’s K-ExaOne was tuned specifically for a certain Korean test set. However, as these are merely online allegations, they do not necessarily imply misconduct in the model development process.

Motif has also faced controversy over the purchase of AAII-related datasets. While Motif acknowledges the purchase, it maintains that it did not engage in so-called “cramming” to inflate evaluation scores and that there is no issue since the model achieved consistently high scores across various categories where the dataset was not required.

◇The Real Battle Begins After Dokpamo… Will They Advance Directly to the Frontier AI Project?

The industry’s attention is shifting beyond the results of the second evaluation to the “post-Dokpamo” phase. This is because there is speculation that, after the government selects the final three models, they will proceed directly to a Frontier-level AI development project starting next year without any additional selection process.

A high-ranking industry official stated, “I’ve heard that the Ministry of Science and ICT plans to select three models and move directly to the Frontier-level project without further selection,” adding, “Whether to launch a separate project next year or proceed with the three selected companies as the core is a matter that requires careful consideration.”

It is highly likely that the final three models will continue to receive computing resources—such as support for 1,000 GPUs—and government R&D funding. However, some point out that since Frontier AI requires not only computing power but also capabilities in cloud, semiconductors, data, and services, an open collaborative ecosystem encompassing the final three companies, the companies that did not make the cut, and related industries is necessary.

◇Extending to “AI for All”… Competition Beyond Models to the Ecosystem

As the government’s AI policy expands beyond a single model to “AI for All,” which aims for a multi-model, multi-agent structure, competition within the ecosystem is expected to intensify. Amid anticipated competition among consortia formed by AI model companies and telecommunications and platform firms, Motif has agreed to participate in the KTCorporation consortium as a model developer.

As government support extends to both Frontier AI and “AI for All,” the focus of competition is expected to shift from model performance to a race to secure computing resources, services, and talent.

In the second evaluation round, attention is focused on whether Motif—which ranked first in the AAII—will secure a spot among the final three companies by demonstrating real-world usability and Korean language capabilities, and whether SKTelecom (017670) and LG Corp. (003550) AI Research Institutes, which were confirmed not to have purchased training data for the AAII, will survive the final competition. The company that successfully translates benchmark performance into actual service competitiveness and secures GPUs, talent, and an ecosystem is expected to take the lead in the domestic AI market following the Dokpamo competition.