Generative AI Security: Risks and Security Controls

Generative AI security is the practice of protecting systems that generate text, code, images, or actions from misuse, compromise, and unintended behavior. It covers the model and everything around it: the prompts users send, the documents a system retrieves, the APIs and providers it depends on, and the tools it can operate.

These systems create distinctive challenges because they take natural-language instructions and data through the same channel. They also produce outputs that other software may trust, and they increasingly take actions on a user's behalf. As a result, one malicious document or one overly permissive integration can turn a helpful assistant into a path for data exposure or unauthorized activity.

This article explains the most important generative AI security risks, the practical controls that reduce them, and how to test, monitor, and govern those controls.

What Is Generative AI Security?

Generative AI security applies established cybersecurity and AI application security principles to systems built on large language models (LLMs) and other generative models. It also adds controls for behaviors that traditional software does not have.

It overlaps with three disciplines:

• Cybersecurity protects infrastructure, identities, networks, and data.

• Application security governs how the software around the model handles input, output, authentication, and authorization.

• Model security concerns the model itself, including training data integrity, weights, and resistance to manipulation or extraction.

The model alone is not the whole attack surface. Risk can enter through user prompts, retrieved content, plugins and APIs, hosting infrastructure, third-party model providers, and connected tools such as email, databases, or ticketing systems. A well-behaved model deployed with broad database access and no output validation can still cause serious harm.

Generative AI security is also distinct from AI ethics, reliability, and privacy compliance, although they overlap. A biased answer or a hallucination is not automatically a security vulnerability. It becomes one when an attacker can induce or exploit it, or when the system acts on it unsafely.

Because risk appears at every stage, security needs attention throughout design, development, deployment, and operation. The CISA and UK NCSC Guidelines for Secure AI System Development (2023) organize their recommendations around exactly these lifecycle stages.

What Are the Main Generative AI Security Risks?

Which risks matter most depends on the system's architecture, deployment context, permissions, and data. A public chatbot with no tool access has a very different profile from an internal agent that can send email and query customer records.

The table below uses the terminology of the OWASP Top 10 for LLM Applications 2025 where a matching category exists.

Indirect prompt injection is not a theoretical concern. Researchers demonstrated it against real LLM-integrated applications in Greshake et al., "Not What You've Signed Up For" (2023). They showed that instructions planted in retrieved web content could manipulate application behavior and support data exfiltration.

Two further 2025 categories deserve attention:

• System prompt leakage (LLM07:2025) occurs when attackers extract hidden instructions. It becomes serious when those prompts contain credentials, internal logic, or access rules that should live in code instead.

• Vector and embedding weaknesses (LLM08:2025) affect retrieval-augmented generation (RAG), where a system retrieves documents from a knowledge base to ground its answers. Weak access controls in the vector store can expose data across users or tenants. Poisoned documents can also inject instructions into every answer that retrieves them.

OWASP also lists Misinformation (LLM09:2025), which matters most where outputs drive decisions without verification. For adversary tactics beyond application-level risks, MITRE ATLAS catalogs real-world techniques used against AI systems.

Which Security Controls Help Protect Generative AI Systems?

Select controls based on the system's threat model, data sensitivity, exposure to untrusted users or content, available permissions, and the potential impact of failure. The safeguards below map directly to the risks above.

Protect Prompts, Inputs, and Model Outputs

Treat every user input and every retrieved document as untrusted. That includes internal content that employees or outsiders can edit.

• Separate instructions from data where possible. Use structured message roles, clear delimiters, and architectures that keep untrusted content out of privileged instruction paths. These reduce risk but do not create a hard boundary.

• Validate outputs before use. Apply schema validation, allow-lists, and context-appropriate encoding before output reaches a browser, database, shell, or other system. Use parameterized queries instead of model-generated SQL strings.

• Never execute model output as trusted code or commands without sandboxing and independent checks.

• Require separate authorization for sensitive operations. A model's decision to act is not authorization.

• Keep secrets and access rules out of system prompts, because those prompts can leak.

System prompts and keyword filters help, but they are not reliable security boundaries on their own. Attackers routinely rephrase, encode, or embed instructions to get around them. OWASP notes that it is unclear whether fool-proof prevention of prompt injection exists given how current models work. The practical goal is layered defense that limits what a successful injection can achieve.

Protect Sensitive Data and Access

Data controls determine how much damage a compromised or manipulated system can do.

• Apply least privilege to the application and its service accounts.

• Enforce authorization at retrieval time. In RAG systems, filter documents by the requesting user's permissions before content reaches the model. Vector databases do not inherit source-system permissions automatically.

• Segment vector stores by tenant or sensitivity level where data must not mix.

• Minimize data sent to models and providers, and review provider terms on retention and training use.

• Manage secrets properly with a secrets manager and regular rotation. Keep credentials out of prompts and code.

• Control logs and retention. Prompts and outputs often contain sensitive data, so they need access controls and redaction where appropriate.

• Encrypt prompts, embeddings, and stored conversations in transit and at rest.

Masking and anonymization reduce exposure but are not absolute. Re-identification through combined fields or context remains possible.

Secure Models, Dependencies, and Integrations

• Assess providers and components before adoption, including models, datasets, plugins, APIs, and libraries.

• Use trusted sources and integrity checks, such as verified repositories, checksums or signatures where available, and safer model serialization formats.

• Maintain an inventory of models, versions, and dependencies, and include AI components in vulnerability management.

• Test model and provider updates before production use, because behavior shifts can weaken existing safeguards.

• Isolate components so a compromised plugin or model cannot reach unrelated systems.

• Review third-party access and changes to deployed systems periodically.

No single measure guarantees model integrity or complete supply-chain security. These controls reduce likelihood and limit impact.

Restrict AI Agent Permissions and Actions

AI agent security needs particular care because agents turn model output into real-world actions. The OWASP GenAI Security Project has published dedicated guidance on agentic AI threats, which reflects how quickly this area is evolving.

• Grant only the tools and permissions each task needs. An agent that summarizes tickets does not need to close them.

• Enforce authorization outside the model. Each tool or API should check whether the acting user and agent are permitted to perform the action, whatever the model requests.

• Prevent "confused deputy" abuse. An agent acting with a privileged service account can be tricked into performing actions the requesting user could not. Where possible, scope tool calls to the end user's own permissions.

• Treat connectors and plugins as trust boundaries. Tool descriptions and tool outputs can carry injected instructions, so vet third-party connectors and validate what they return.

• Limit multi-agent trust. One agent's output is untrusted input to the next.

• Use human approval for sensitive or irreversible actions where appropriate, such as payments, external messages, or deletions.

• Add validation and limits, including transaction caps, rate limits, allow-listed destinations, and execution sandboxes.

• Keep audit trails and a stop mechanism so unsafe activity can be traced and suspended quickly.

Human approval is one safeguard, not a substitute for technical access controls. Reviewers can be rushed, misled by a plausible summary, or overwhelmed by volume.

Build Your AI Security and Governance Knowledge
Applying these controls well requires understanding both technical risks and organizational accountability. AGC's AI Security Governance & Compliance Fundamentals course provides a structured foundation in AI security risks, governance responsibilities, and compliance considerations.

Risk-to-Control-to-Test Matrix

This matrix connects each risk to its primary safeguards and to a concrete way of verifying that those safeguards work.

How to Implement Generative AI Security Controls

This workflow puts the controls into practice. For a full methodology, see How to Conduct an AI Risk Assessment.

Step 1: Inventory the System

Document the model and version, use case, data sources, integrations, providers, users, and every action the system can take. You cannot secure components you haven't identified.

Step 2: Map Threats and Trust Boundaries

Identify who can supply input, directly or through documents, what data the system can reach, which components are trusted, and what it can execute. Indirect injection paths often surface here.

Step 3: Prioritize Risks

Assess realistic attack paths against existing safeguards. An injection that only changes a chatbot's tone matters less than one that can email customer data externally.

Step 4: Select and Assign Controls

Define safeguards, owners, implementation requirements, and acceptance criteria, such as "retrieval returns only documents the requesting user can access."

Step 5: Test Control Effectiveness

Use the matrix above to verify injection resistance, data-access restrictions, output handling, and action limits against those acceptance criteria.

Step 6: Monitor and Improve

Track security events, exceptions, incidents, and residual risks. Reassess whenever models, data sources, tools, or integrations change materially.

How to Test and Monitor Generative AI Security

AI security testing confirms whether controls work in practice. It should be authorized, scoped, and proportionate to the system's risk.

• Threat-model-driven testing verifies that each identified attack path is blocked or contained.

• Adversarial testing and red-teaming have testers try to cause policy violations, data leakage, or tool misuse in scenarios realistic for the use case. MITRE ATLAS can help structure adversary scenarios.

• Direct and indirect prompt injection testing should include poisoned documents, emails, and web pages, not only malicious chat input.

• Retrieval testing confirms that permission filters hold across users, roles, and tenants.

• Output and tool-execution testing confirms that malicious payloads are neutralized and agents cannot exceed their scope.

• Access-control and integration testing applies conventional application security methods to authentication, token scope, and APIs.

Monitoring should flag unusual query patterns, policy violations, repeated access failures, abnormal token consumption, and unexpected agent actions. Route these signals into existing incident escalation, containment, recovery, and retesting processes.

A passed test shows only that controls held against the scenarios tried. It does not guarantee immunity from new techniques. Retest after material changes to models, prompts, datasets, or integrations, and on a regular cadence matched to risk.

How AI Security Controls Fit into AI Security Governance

Technical controls work only when someone owns them. AI security governance provides that accountability.

• Assign responsibilities across AI engineering, IT, information security, risk management, compliance, and business owners.

• Set lifecycle security requirements for procurement, development, deployment, and operation.

• Document control owners, evidence, exceptions, and residual risks so decisions can be reviewed and audited.

• Integrate with existing processes, including change management, vulnerability management, and incident response.

• Use recognized guidance. The NIST Generative AI Profile (NIST AI 600-1, 2024), a companion to the NIST AI Risk Management Framework, and the OWASP GenAI Security Project provide practical structure. Organizations building a formal management system may also align with ISO/IEC 42001.

• Review third-party AI security and changes in the AI supply chain on an ongoing basis.

Keep four categories distinct. Technical recommendations describe good practice. Voluntary frameworks such as NIST, OWASP, and MITRE guidance are not laws. Contractual obligations come from customer and supplier agreements. Applicable laws vary by jurisdiction and sector. For example, the EU AI Act (Regulation (EU) 2024/1689) includes accuracy, robustness, and cybersecurity requirements for high-risk AI systems, and whether they apply depends on the system and its use. Adopting a framework does not by itself establish legal compliance, so seek qualified legal advice for specific obligations. For enterprise-level processes, see AI Risk Management: Complete Guide to Identifying, Assessing & Managing AI Risks.

Conclusion: Build Security into the AI Lifecycle

Generative AI security risks center on manipulated inputs, exposed data, untrusted outputs, compromised components, over-permissioned agents, and resource abuse. No single safeguard addresses them all. Effective protection combines layered controls, least privilege, careful data handling, authorization enforced outside the model, ongoing testing and monitoring, and clear accountability.

These systems also change. New models, data sources, and tools alter the attack surface, so security must evolve alongside them rather than being completed once at launch.

Build Your AI Security and Governance Knowledge

If you deploy, secure, or oversee AI systems, structured training can help you connect technical safeguards with governance and compliance responsibilities. AGC's AI Security Governance & Compliance Fundamentals course helps you build foundational knowledge of AI security risks, governance practices, and compliance considerations that you can apply in your organization.