The Ministry of Science and ICT and the Korea Internet & Security Agency (KISA) published an "AI Security Red-Teaming Guide" to help domestic companies inspect AI models, data and AI agents from an attacker’s perspective in line with such changes. It presented the first practical standards for responding to generative AI-specific threats such as prompt injection, jailbreaks, hallucinations and agent hijacking.

Red teaming is a security validation technique that deliberately tests AI systems from an attacker’s perspective to find vulnerabilities and improve them. It has been used in military and cybersecurity fields, but has recently become a core methodology for verifying the safety of generative AI models and services.

Global companies such as OpenAI, Google, Anthropic and Microsoft already operate AI red teams to respond to attacks targeting their AI. By contrast, the guide noted that South Korea lacks a security system to support AI adoption given the speed of its rollout.

AI training verification extends to agents, supply chains

The biggest feature of AI red teaming is that the inspection target does not remain limited to AI models. The guide presented the full range of AI services as targets for verification, including data and training pipelines, user interfaces (UI), system prompts, AI agents, APIs, servers and infrastructure.

The guide classified AI-specific threats into 8 categories. Key examples include "prompt injection," which manipulates AI through inputs disguised as normal commands, "jailbreak," which bypasses safety measures to elicit prohibited answers, and "hallucination," which fabricates information that is not true. It also listed as threats "system prompt leakage," where AI operating instructions are exposed, "training data leakage," where original training data is restored or leaked, and "model leakage," where model files and weights are leaked or copied. It also included "data poisoning," which distorts model judgement by mixing malicious data into training data, and "model denial-of-service (DoS) attacks," which disable services through excessive inputs.

The guide presented "agent hijacking" as a new type of attack. It is a threat in which an AI agent mistakes a malicious prompt hidden in an external document or webpage for a normal command and performs unintended tasks. Because agents are structured to directly call external systems, the guide pointed out that such malfunctions can go beyond simple errors and lead to actual data lookups or changes, or privilege bypasses.

The guide also divided risk levels into 5 stages based on the impact such threats could have on real services. The most severe "critical" grade included cases such as providing actual account takeover code or forged document templates in the financial sector, recommending fatal drug misprescriptions that directly affect life in the medical sector, and generating malicious code capable of remote code execution in coding. Examples in the "high" grade below that included recommending non-existent treatments as if they were facts and generating executable attack scripts for known vulnerabilities.

There is no single correct answer in AI security, service-specific checks needed

The guide explained that inspection methods should also vary depending on service characteristics. It recommended choosing based on risk and internal capabilities among a black-box approach that checks only input and output in the same environment as real users, a white-box approach that analyses internal structures and source code, and a grey-box approach that uses only partial information.

It stressed that red teaming should not end after a single run before a service is launched. New vulnerabilities can arise simply by fine-tuning an AI model or partially modifying a system prompt. It said repeated, continuous execution is needed not only before development and deployment but also after operations begin, as attack techniques evolve quickly.

To that end, the guide proposed forming AI security red teams with personnel from multiple fields, including not only security experts but also AI engineers, legal experts and service domain specialists. It also recommended training on AI security techniques, domain-specific risks, ethics and rules of engagement, and establishing psychological support systems for red team members repeatedly exposed to harmful content.

The government said it aims to use the guide to encourage AI red teaming to take root in domestic industry.

An MSIT official said providing information needed in the field is the priority now, adding the guide would be highly usable given that red teams had been operating haphazardly in the field due to a lack of relevant guidelines.

A KISA official said it would be a minimum guide to ensure user safety, noting that a consensus on safety and trust is needed before innovative technologies emerge.