[Edaily Reporter Kim Hyun-ah ] The specific reasons why Motif Technologies—which had ranked first in Korea and 10th globally in the AI Model Evaluation (AAII) conducted by the global benchmarking organization Artificial Analysis—was eliminated in the second phase of the Independent AI Foundation Model Project (Dokpamo) evaluation have been disclosed.

Although Motif received the highest score in the global benchmark, it fell behind SKTelecom(017670)and the AI Research Institute at LG Corp.(003550) in the NIA benchmark, expert evaluations, and real-user evaluations.

A high-ranking official from the Ministry of Science and ICT explained in a phone call with E-Daily regarding Motif’s elimination, “It wasn’t simply a matter of low practicality; the scores were generally low in both expertise and usability,” adding, “It is true that the performance fell short.”

On the 20th, the Ministry of Science and ICT disclosed the detailed evaluation criteria for the second phase of the Dokpamo evaluation, along with the top-ranked companies for each category. This disclosure—which revealed the top-ranked companies and their scores for each evaluation category, similar to the first phase—came in response to controversy over the transparency of the evaluation, as the ministry had not released detailed scores following the recent announcement of the evaluation results.

This evaluation was scored out of a total of 100 points, consisting of 40 points for benchmarks, 35 points for expert evaluations, and 25 points for user evaluations.

Most notably, the top-ranked company differed by evaluation category. Motif took first place in the AAII Global Benchmark, while SKTelecom ranked first in the NIA Benchmark and the AI expert user evaluation. LG Corp. took first place in both the expert evaluation and the general public evaluation.

In the 25-point Global AAII Benchmark category, Motif took first place with 11.9 points. In contrast, SKTelecom scored the highest in the 15-point NIA Benchmark category with 13.4 points.

In the 35-point expert evaluation, LG Corp. AI Research took first place with 29.5 points. In the 15-point AI expert user evaluation, SKTelecom scored the highest with 11.6 points, while in the 10-point general public evaluation, LG Corp. AI Research ranked first with 7.6 points.

While Motif demonstrated its strengths in the global public benchmark, SKTelecom and LG Corp. AI Research took the lead in the domestic benchmark as well as in the expert and actual user evaluations.

Regarding Motif’s evaluation results, a high-ranking official from the Ministry of Science and ICT stated, “It wasn’t a matter of practicality; rather, both expertise and usability were somewhat lacking,” adding, “While it’s true that performance fell short, the point difference isn’t significant.”

Motif scored 11.9 on the AAII benchmark—the highest possible score out of 25—but in the NIA benchmark, SKTelecom took first place with a score of 13.4.

The NIA benchmark evaluates seven areas: mathematics, knowledge, long-text comprehension, safety, reliability, Korean language, and task execution. It separately verified the model’s performance in the domestic environment, which is difficult to capture using a single global public metric.

Motif also trailed behind LG Corp. AI Research in the expert evaluation. The expert evaluation involved external experts—including three from industry, five from academia, and two from the research sector—who conducted written evaluations and Q&A sessions over approximately five days to assess development strategies and technologies, development achievements and plans, ecosystem impact, and plans for contribution.

In particular, 10 out of 35 points were allocated to development strategy and technology. This category included the excellence and innovation of the developed technology, its technical originality, the level of performance improvement, resource utilization and processing efficiency, and technologies for developing safe and reliable AI models.

The remaining 10 points were allocated to development achievements and plans, and 15 points were allocated to ecosystem and global impact.

LG Corp. AI Research took first place in this expert evaluation with a score of 29.5 points. The average score for the four elite teams was 28.75 points.

Motif also failed to take first place in the user evaluation. In the AI expert user evaluation—composed of representatives from AI startups and others—SKTelecom ranked first with 11.6 points, while LG Corp. scored the highest among the general public with 7.6 points.

The Ministry of Science and ICT explained that it guided the user evaluations to focus on the content and quality of the content generated by the AI models, rather than simply evaluating website design or UI/UX.

A total of 49 AI experts and 185 members of the general public used the actual AI services and assigned scores using an absolute evaluation method.

These results demonstrate that the final rankings were not determined solely by specific benchmark performance. Even if Motif had ranked first in global benchmarks, the evaluation structure allowed for it to be eliminated in the final results, which combined domestic performance verification, expert assessments of technical and business viability, and actual user evaluations.

The Ministry of Science and ICT also separately reviewed the minimum criteria for originality in this second evaluation. The assessment included whether the models met the standards for original AI models from technical, policy, and ethical perspectives.

Regarding Motif, considering that it was a new participating team, the evaluation of its ecosystem ripple effects included the field deployment track record and plans for existing AI models. The Ministry explained that while the three existing elite teams were evaluated based on the field deployment track record and plans for their proprietary AI models and derivative models, Motif was evaluated by including the field deployment track record and plans for existing AI models as well.