[Edaily Reporter Kim Hyun-ah ] The specific reasons why Motif Technologies—which had ranked first in Korea and 10th globally in the AI Model Evaluation (AAII) conducted by the global benchmarking organization Artificial Analysis—was eliminated in the second phase of the Independent AI Foundation Model Project (Dokpamo) evaluation have been disclosed.

Although Motif received the highest score in the global benchmark, it fell behind SKTelecom(017670)and the AI Research Institute at LG Corp.(003550) in the NIA benchmark, expert evaluations, and real-user evaluations.

A high-ranking official from the Ministry of Science and ICT explained to E-Daily regarding Motif’s elimination, “It wasn’t simply a matter of low practicality; the scores were generally low in both expertise and usability,” adding, “It is true that the performance fell short.”

On the 20th, the Ministry of Science and ICT disclosed the detailed evaluation criteria for the second phase of the Dokpamo evaluation, along with the top-ranked companies in each category. This disclosure came after controversy arose over the transparency of the evaluation following the recent announcement of results, as detailed scores had not been released; consequently, the Ministry decided to disclose the top-ranked companies and scores for each evaluation category, consistent with the approach taken in the first phase.

This evaluation was scored out of a total of 100 points, consisting of 40 points for benchmarks, 35 points for expert evaluations, and 25 points for user evaluations.

Most notably, the top-ranked companies differed by evaluation category. Motif took first place in the AAII Global Benchmark, while SKTelecom topped the NIA Benchmark and the AI expert user evaluation. LG Corp. came in first in both the expert evaluation and the general public evaluation.

In the 25-point Global AAII Benchmark category, Motif took first place with 11.9 points. In contrast, SKTelecom scored the highest—13.4 points—in the 15-point NIA Benchmark category.

In the 35-point expert evaluation, LG Corp. AI Research took first place with 29.5 points. In the 15-point AI expert user evaluation, SKTelecom scored the highest with 11.6 points, while in the 10-point general public evaluation, LG Corp. AI Research ranked first with 7.6 points.

While Motif demonstrated its strengths in the global public benchmark, SKTelecom and LG Corp. AI Research led in the domestic benchmark as well as in the expert and actual user evaluations.

Regarding Motif’s evaluation results, a high-ranking official from the Ministry of Science and ICT stated, “It wasn’t a matter of practicality; rather, both expertise and usability were somewhat lacking,” adding, “It’s true that the performance didn’t measure up.”

This explanation differs somewhat from the Ministry’s initial justification for Motif’s disqualification, which had emphasized “practicality” and “real-world usability.” Based solely on the briefing at the time, there was room to interpret that Motif—which had achieved high scores in global benchmarks—was eliminated based on the applicability criterion rather than technical capability; however, with the release of the detailed results, it has been confirmed that Motif also lacked competitiveness in terms of expertise and user evaluations.

In particular, while Motif scored 11.9 on the AAII benchmark—the highest possible score out of 25—SKTelecom took first place in the NIA benchmark with a score of 13.4.

The NIA benchmark evaluates seven areas: mathematics, knowledge, long-text comprehension, safety, reliability, Korean language, and task execution. It separately verified the model’s performance in the domestic environment—a factor that is difficult to capture using a single global public metric.

Motif also trailed behind LG Corp. AI Research in the expert evaluation. The expert evaluation involved external experts—three from industry, five from academia, and two from the research sector—who conducted written evaluations and Q&A sessions over approximately five days to assess development strategies and technologies, development achievements and plans, ecosystem impact, and contribution plans.

In particular, 10 out of 35 points were allocated to development strategy and technology. This included the excellence and innovation of the developed technology, its technical originality, the level of performance improvement, resource utilization and processing efficiency, and technologies for developing safe and reliable AI models.

The remaining 10 points were allocated to development achievements and plans, and 15 points were allocated to ecosystem and global impact.

LG Corp. AI Research took first place in this expert evaluation with a score of 29.5 points. The average score for the four elite teams was 28.75 points.

Motif also failed to take first place in the user evaluation. In the AI expert user evaluation—composed of representatives from AI startups and others—SKTelecom ranked first with 11.6 points, while LG Corp. scored the highest among the general public with 7.6 points.

The Ministry of Science and ICT explained that it guided the user evaluations to focus on the content and quality of the content generated by AI models, rather than simply evaluating website design or UI/UX.

A total of 49 AI experts and 185 members of the general public used the actual AI services and assigned scores using an absolute evaluation method.

These results demonstrate that the final rankings were not determined solely by performance on specific benchmarks. Even if Motif had ranked first in global benchmarks, the evaluation structure allowed for it to be eliminated in the final results, which combined domestic performance verification, expert assessments of technical and business viability, and actual user evaluations.

The Ministry of Science and ICT also separately reviewed the minimum criteria for originality in this second evaluation. The assessment included whether the models met the standards for original AI models from technical, policy, and ethical perspectives.

Regarding Motif, considering that it was a new participating team, the evaluation of its ecosystem ripple effect included the field deployment track record and plans of existing AI models. The Ministry explained that while the three existing elite teams were evaluated based on the field deployment track record and plans of their proprietary AI models and derivative models, Motif was evaluated by including the field deployment track record and plans of existing AI models as well.

A high-ranking official at the Ministry of Science and ICT said, “If the performance had been decent, I actually had high expectations,” adding, “There was a sense of anticipation that if a startup could develop a model this well, large corporations would be able to offer services based on it, creating a positive cycle.”

The official went on to explain that Motif’s performance in this evaluation fell short of initial expectations.

With the release of these detailed results, the controversy surrounding the second round of the Dokpamo evaluation has entered a new phase. While the initial controversy centered on questions such as “Why was Motif, which ranked first in the AAII, eliminated?” and “Did the government evaluate only practicality rather than technical capability?”, a comprehensive review of the data released by the Ministry of Science and ICT narrows the actual issue down to the scoring structure used to combine models whose strengths varied across different evaluation categories.

Motif ranked first in the global benchmark, SKTelecom ranked first in the NIA and AI expert user evaluations, and LG Corp. AI Research ranked first in the expert and general public evaluations, respectively.

Accordingly, the Ministry of Science and ICT explained that in the comprehensive evaluation out of 100 points, the final selection was not determined solely by global benchmark scores, but also took into account expertise, practical usability, and the ripple effect on the ecosystem.

However, it seems difficult to avoid criticism that the government fueled unnecessary misunderstandings by failing to sufficiently explain the key evaluation finding—Motif’s “lack of expertise and performance”—during its initial briefing. It remains to be seen whether the release of these detailed results will serve as an opportunity to restore confidence in the Dokpamo evaluation.