OpenAI Details Security Strategies for AI Agents Against Prompt Injection Attacks
March 11, 2026 — OpenAI has published a comprehensive overview of its approach to securing AI agents against prompt injection attacks, drawing parallels between defending artificial intelligence systems and mitigating social engineering risks targeting human employees.
The technical documentation reveals that as AI agents become increasingly capable of browsing the web, retrieving information, and executing actions on behalf of users, they have also become attractive targets for manipulation attempts. These attacks, commonly referred to as prompt injection, involve placing malicious instructions within external content with the goal of compelling AI systems to perform actions users never intended.
The Evolution of Prompt Injection Attacks
According to the analysis, prompt injection techniques have undergone significant evolution. Early versions of such attacks were relatively straightforward, such as editing Wikipedia articles to include direct instructions for AI agents. At that time, models without training experience in adversarial environments would often follow those instructions without question.
However, as AI models have grown more sophisticated, they have become less vulnerable to these simple suggestions. In response, attackers have adapted their methods to incorporate elements of social engineering, making the attacks more sophisticated and harder to detect.
A notable example documented from 2025 involved a detailed email crafted to manipulate AI assistants. The attack, reported to OpenAI by external security researchers, succeeded approximately 50 percent of the time when users prompted their assistants to conduct deep research on emails. The malicious email appeared to contain legitimate work requests, including instructions about reviewing employee data, finalizing role descriptions, and coordinating with finance departments—contextual elements designed to appear trustworthy.
Limitations of Current Defense Mechanisms
The security analysis acknowledges that within the broader AI security ecosystem, techniques such as “AI firewalling” have become common recommendations. These systems attempt to classify inputs as either malicious prompt injection or legitimate content by serving as intermediaries between AI agents and external data sources.
However, the documentation notes that these fully developed attacks typically evade such classification systems. For these defensive mechanisms, detecting a malicious input presents the same fundamental challenge as identifying deception or misinformation—often without access to the necessary contextual information to make accurate determinations.
A Social Engineering Framework for AI Security
The company’s security team has adopted a perspective that treats prompt injection attacks with social engineering elements as analogous to social engineering risks that human workers face in other domains. Rather than seeking to perfectly identify every malicious input, this approach focuses on designing agents and systems so that the potential impact of successful manipulation remains constrained.
This framework positions the AI agent within a three-actor system similar to that of a customer service representative. In this scenario, the agent acts on behalf of its employer but remains continuously exposed to external input that may attempt to mislead it. Whether the agent is human or artificial, limitations must be placed on its capabilities to manage the downside risk inherent in operating within potentially adversarial environments.
The documentation draws parallels to real-world customer support systems where agents possess the authority to issue refunds or gift cards. Corporations must trust that agents grant such benefits for legitimate reasons, while simultaneously interacting with third parties who may attempt to mislead them or place them under duress. Deterministic systems can limit the number of refunds permitted, flag potential phishing attempts, and implement other mitigations to reduce the impact of compromising an individual agent.
Implementation in ChatGPT and Related Products
OpenAI has combined this social engineering model with more traditional security engineering approaches, including source-sink analysis. Within this framework, attackers require both a source—meaning a way to influence the system—and a sink, which refers to a capability that becomes dangerous when misused. For agentic systems, this typically involves combining untrusted external content with actions such as transmitting information to third parties, following links, or interacting with tools.
The company has established a core security expectation for users: potentially dangerous actions or transmissions of potentially sensitive information should not occur silently or without appropriate safeguards.
The analysis indicates that attacks against ChatGPT most frequently attempt to convince the assistant to extract secret information from a conversation and transmit it to a malicious third party. In most documented cases, these attacks fail due to safety training that causes the agent to refuse such requests.
For cases where the agent does become convinced, OpenAI has developed a mitigation strategy called Safe Url. This mechanism is designed to detect when information the assistant has learned during the conversation would be transmitted to a third party. In these uncommon scenarios, the system either displays the information that would be transmitted to the user for confirmation or blocks the transmission and instructs the agent to pursue an alternative approach to fulfill the user’s request.
This same protective mechanism applies to navigation and bookmark features in Atlas, as well as search and navigation functions in Deep Research. ChatGPT Canvas and ChatGPT Apps employ similar approaches, allowing agents to create and use functional applications that operate within sandboxed environments capable of detecting unexpected communications and requesting user consent.
The company stated that this security mindset has informed the development of a robust suite of countermeasures designed to uphold the security expectations of users interacting with AI agents.
Source: https://openai.com/index/designing-agents-to-resist-prompt-injection/
