[Edaily Reporter Kim Hyun-ah ] Kakao(035720)’s lightweight language model (SLM) “Kanana-2” series scored highly in a safety evaluation comparing it to global open-source models of similar scale.

Kakao announced on the 18th that it had evaluated the harmfulness and bias of “Kanana-2-1.3B-Instruct” and “Kanana-2-3B-Instruct”—which were released as open source on the 28th of last month via its in-house “AI Safety Evaluation Platform”—and found that both models outperformed the comparators, Google’s “Gemma” and Alibaba’s “Qwen” in overall scores.

This evaluation utilized “AssurAI,” a Korean-language-specific safety benchmark jointly developed last November by the KoreaInformation&Communication Technology Association (TTA), KAIST, and Kakao as part of a project by the Ministry of Science and ICT.

AssurAI is a safety evaluation system designed to reflect Korea’s social and cultural context. It can measure risk factors in AI models under various conditions, including not only text but also multimodal environments such as images and audio, as well as AI service usage scenarios and malicious prompt scenarios.

It consists of a total of 35 risk categories and 9,560 evaluation items. Kakao reorganized these into five major categories—social risks, sexual content and child protection, crime and illegal activities, violence, and rights violations—for evaluation. The “LLM-as-a-Judge” method, in which another AI evaluates a model’s responses based on predefined criteria, was also applied.

The evaluation results showed that Kanana-2-1.3B-Instruct achieved an overall score of 0.70, outperforming Gemma (0.68) and Qwen (0.58). Kanana-2-3B-Instruct also scored 0.70, outperforming Gemma (0.66) and Qwen (0.62).

It also demonstrated strengths in detailed evaluations. The Kanana-2 1.3B model received high marks in the area of crime and illegal activities, while the 3B model scored higher than the comparison models in the areas of sexual content, child protection, and rights violations.

In particular, Kanana-2 was developed as a lightweight model targeting mobile environments. Kakao developed the 1.3B model by applying pruning—a process that reduces parameters—to the 3B model. The company explained that it enhanced usability in mobile and agent services with heavy Korean input and output by applying a Korean-specialized tokenizer and a sliding attention-based effective context processing technique.

Ha Jeong-woo, former Senior Secretary for AI Future Planning at the Office of the President, also praised the development of the domestic AI ecosystem last month on his Facebook page, mentioning Kakao’s release of Kanana-2.

Former Senior Secretary Ha stated, “Kakao has been steadily researching and developing Kanana, a foundation model (FM), and has now released the mobile-optimized FM Kanana-2-SLM in 1.3B and 3B sizes,” adding, “It is competitive compared to other FMs in its class and incorporates a Korean-specialized tokenizer and a sliding attention-based effective context processing technique.”

He continued, “I recommend that anyone in need of a small, fast Agent AI service with heavy Korean input and output give it a try,” adding, “I am proud to see that domestic AI companies—not just the ‘Big Four’—are actively releasing their models, and that Korea’s AI technology ecosystem is steadily advancing.” He concluded by saying, “I wholeheartedly support all Korean AI companies.”

Kakao plans to use this evaluation as an opportunity to institutionalize ongoing pre-release safety verification for its in-house AI models. The company intends to expand the scope of evaluation beyond text-based language models to include multimodal models and agentic AI in the future.

Kim Kyung-hoon, Kakao’s AI Safety Leader, stated, “This is an example of us proactively checking the safety of open-source models through our internal verification system and disclosing the results,” adding, “In accordance with the principles of responsible AI development, we will establish safety verification as a core step in the model release process.”

Meanwhile, Kakao’s in-house AI model, “Kanana,” received a higher rating than overseas models of similar scale—Meta’s “Llama” and Mistral’s “Mistral”—in South Korea’s first AI safety evaluation, conducted last December by the Ministry of Science and ICT, the AI Safety Research Institute, and the TTA.