[Edaily Reporter Kim Hyun-ah ] The Ministry of Science and ICT (Minister Bae Kyung-hoon) has stepped forward to emphasize the fairness and transparency of the evaluation criteria and process in response to controversy surrounding the results of the second-stage evaluation of “Independent AI Foundation Models (DOKPAMO).”

In particular, the Ministry explained that this evaluation was not conducted based on criteria unilaterally set by the government, but rather operated under a “moving target” approach, in which evaluation goals and criteria are adjusted every six months to keep pace with rapid changes in AI technology. It added that the criteria for the second evaluation were also established following multiple consultations with four elite teams.

Motif Technologies, which ranked first among participating companies and tenth globally in the AAII benchmark—a trusted standard in the global market—was eliminated in the second round of evaluations. After the company demanded the release of detailed scores and other elite teams agreed, the Ministry of Science and ICT disclosed the scores for each evaluation category—which had not been made public until then—on the night of the 27th.

The Ministry of Science and ICT stated, “The ‘Dokpamo’ program is not intended to be a simple ranking competition or to select teams for elimination; its purpose is to enable our AI companies to grow beyond competition and drive the qualitative growth and expansion of the domestic AI ecosystem.”

It also explained, “While striving to minimize any unexpected direct or indirect harm to certain companies resulting from the disclosure of evaluation results, we have carefully determined the scope of disclosure by maintaining a balanced consideration of the values of fairness and transparency in the evaluation process and outcomes.”

The Ministry of Science and ICT stated that, in light of the rapid evolution of AI technology, it is applying a “moving target” approach that revises AI model development goals every six months. Accordingly, the criteria and methods for each phase of evaluation are established through consultation with the elite team on a case-by-case basis.

This second-phase evaluation was conducted on a 100-point scale, comprising 40 points for benchmark evaluation, 35 points for expert evaluation, and 25 points for user evaluation.

The benchmark evaluation consisted of 25 points from the AAII assessment by Artificial Analysis, a global AI evaluation platform, and 15 points from the National Information Society Agency (NIA). The expert evaluation assessed development strategy and technology, development achievements and plans, and anticipated impact and contribution plans, while the user evaluation included assessments by AI experts and the general public.

The final scores were as follows: SKTelecom(017670) with 70.6 points, Upstage with 69.9 points, LG Corp.(003550) AI Research Institute with 69.0 points, and Motif with 65.8 points. Consequently, three elite teams—SKTelecom, Upstage, and LG Corp. AI Research Institute—advanced to the next stage, while Motif was eliminated.

Looking at the details, the results varied significantly across evaluation categories. In the AAII—a global benchmark—Motif took first place with 11.9 points. In contrast, LG Corp. AI Research ranked first in the expert evaluation with 29.5 points; SKTelecom received the highest score in the AI expert user evaluation with 11.6 points; and LG Corp. AI Research received the highest score in the general public evaluation with 7.6 points.

The Ministry of Science and ICT stated regarding the criteria and methodology for this second evaluation: “These were established after multiple consultations with the elite teams. Following the official announcement of the final plan prior to the evaluation, there were no objections from the elite teams, and the evaluation was conducted based on this.”

The evaluation system also changed compared to the first round. The ministry explained that it newly adopted the AAII, which enjoys high global credibility; in the expert evaluation, it increased the weight given to actual development achievements and the ripple effects on domestic and international ecosystems—in addition to development strategies—and added a public evaluation for the first time.

Regarding this, the Ministry of Science and ICT stated, “The significance of the phased evaluation lies not merely in narrowing down and selecting teams through simple competition among elite teams, but in the fact that the government actively designed and evolved the evaluation criteria to encourage elite teams to strengthen their capabilities in developing world-class AI models, integrate and disseminate proprietary AI models into the domestic AI ecosystem, and develop AI models that meet public expectations.”

The expert evaluation involved 10 external experts: 3 from industry, 5 from academia, and 2 from the research sector. Based on the evaluation documents submitted by the elite teams, the panel conducted a written evaluation and a question-and-answer session. The final score was calculated as the arithmetic mean of the remaining scores after excluding the highest and lowest scores from each committee member.

User evaluations were also conducted as absolute evaluations rather than relative comparisons among the elite teams. Forty-nine AI expert users and 185 members of the general public used each model directly and rated them on a scale of 1 to 5.

The Ministry of Science and ICT stated, “This evaluation was designed to go beyond the performance demonstrated by simple AI models and comprehensively assess the usability and perceived effectiveness experienced by a diverse range of actual users when using the AI models directly.”

However, controversy is expected to persist regarding the impact of the evaluation scoring system and methodology on the results, given that Motif—which ranked first in the AAII, a global performance benchmark—was ultimately eliminated after receiving relatively low scores in the expert and user evaluations.

The Ministry of Science and ICT gave a positive assessment of Motif’s technological capabilities themselves. It explained that Motif’s “Motif 3” recorded the highest performance among AI models developed by countries other than the U.S. and China in the AAII, and demonstrated a level on par with global Big Tech companies in high-difficulty benchmarks such as Agent AI tasks and coding.

The Ministry stated, “Motif Technologies’ elite team achieved the world’s best performance among AI models developed by countries other than the U.S. and China at the AAII, demonstrating a high level of capability on par with the world’s leading big tech companies in high-difficulty benchmark areas such as Agent AI and coding. Given their significant achievements in developing proprietary AI models, they are expected to continue contributing to the domestic AI ecosystem in various ways.”

Support for high-performance GPUs will also be expanded for the three elite teams that have advanced to the next stage. Based on the B200 standard, the number of GPUs provided increased from approximately 500 in the second half of last year to 768 in the first half of this year, and the plan is to expand this to around 1,000 in the second half of this year. This amounts to a budget of 40 billion won each for SKTelecom, Upstage, and LG Corp. AI Research.

Considering the recent intensification of AI model competition, centered on the United States and China, and the emergence of AI models as strategic assets that determine national security and industrial competitiveness, the Ministry of Science and ICT is also discussing a new support system for the development of top-tier proprietary AI models with relevant government agencies.

The Ministry stated, “We deeply recognize the need to develop top-tier, independent AI models and are currently discussing a new support system that goes beyond existing approaches with relevant ministries. We plan to prepare and announce specific details at a later date.”