Can an AI agent's resistance to hidden attacks be credited to its model alone? We kept the model fixed and changed only the software around it. How often the attack succeeded moved by up to 45 percentage points.
An agent is a language model that can use tools: read files, send e-mails, move money. AgentDojo, a public test bed for agent security, gives the agent an ordinary request and a fake bank account. In one task the conversation goes like this:
Instructions planted in data that an agent reads are called a prompt injection. Security results for agents are usually reported per model: "model X follows injected instructions Y% of the time." But a model never acts on its own. A program called the harness runs the agent: it builds the conversation, calls the tools and decides when to ask the human. Before the model ever reads the attack, the harness has already made two choices:
system, user, assistant or tool. The harness decides whether the bill's text is labelled as a tool message or as a user message. The agents of two popular frameworks, LlamaIndex's ReAct agent and smolagents' ToolCallingAgent, label it user by default.RoleCall keeps the model weights fixed and changes only these two harness choices. If the model alone set its security, nothing should move.
user messages (the default) instead of tool messages.We ran the installed LlamaIndex and smolagents agents on AgentDojo twice, once with each framework's default and once with its documented tool-role option. For Llama-3.3-70B the default raised attack success by 29.3 points in LlamaIndex and 45.0 points in smolagents. A framework setting relabels every tool output, so we also isolated a single message: we replayed each episode up to the first tool output that contained the attack, then continued once with that one message labelled tool and once labelled user, every other byte identical. That one label raised Llama's attack success by 24 to 27 points. gpt-oss-120b barely moved. Qwen3-32B is a built-in control: its chat template already shows tool messages to the model as user turns, so the switch should change little, and it did.
| Model | Setting | Labelled tool | Labelled user | Change |
|---|---|---|---|---|
| Installed agents: tool-role option vs. framework default | ||||
| Llama-3.3-70B | LlamaIndex | 65.9 | 95.2 | +29.3 |
| Llama-3.3-70B | smolagents | 39.0 | 83.9 | +45.0 |
| gpt-oss-120b | smolagents | 83.1 | 84.3 | +1.2 |
| Qwen3-32B (control) | LlamaIndex | 61.0 | 60.2 | −0.8 |
| Qwen3-32B (control) | smolagents | 68.3 | 65.1 | −3.2 |
| Same history, only the label of the one attacked message changes | ||||
| Llama-3.3-70B | attack text with no sender | 56.7 | 83.9 | +27.2 |
| Llama-3.3-70B | attack text signed as the user | 68.8 | 92.4 | +23.7 |
| gpt-oss-120b | attack text with no sender | 80.2 | 77.4 | −2.8 |
| gpt-oss-120b | attack text signed as the user | 83.5 | 83.5 | 0.0 |
Attack success rate (%). Top: 249 attacked task pairs from 37 user tasks in the banking and Slack suites. Bottom: episodes branched at the first attacked message (224 and 212 pairs). Bold changes have 95% intervals that exclude zero.
In the OpenHands agent framework the model can attach a risk label to every call it proposes, and with the confirmation policy turned on the harness asks the user only when that label is HIGH or missing. We copied this mechanism onto AgentDojo's tools and used the same risk policy, which says to rate HIGH whenever sensitive data leaves the environment. Then the same model rated the same proposed call from three positions: acting (the label inside its own call), asked (a follow-up question in the same conversation) and reviewing (a fresh conversation in which the transcript is shown as another assistant's work). On the first proposed call in each episode that would send money or data to the attacker, the acting label said HIGH in 4.4% (gpt-oss-120b), 42.6% (Qwen3-32B) and 33.0% (Llama-3.3-70B) of episodes. Reviewing the same calls, the same models said HIGH in 76.7%, 82.4% and 88.4%. Simply asking in the same conversation already raised the rating in every model.
For every attacked episode we also ran an authorised twin: the same task, no attack, and the user's request itself asks for the attacker's transfer or e-mail. A good rater should say HIGH on the attack and not on the twin. Reviewers rated 63–88% of these authorised sends HIGH as well. That is consistent with the policy, which escalates any data leaving the environment, but it means the level of a rating says little. We therefore measured separation, the HIGH rate on attacks minus the HIGH rate on twins. Only 2 of the 9 model-and-position combinations separate clearly: Llama-3.3-70B while acting and Qwen3-32B while reviewing.
| Position | gpt-oss-120b | Qwen3-32B | Llama-3.3-70B |
|---|---|---|---|
| Acting (label inside its own call) | 3.6 / 1.2 | 41.4 / 44.6 | 37.0 / 16.4 |
| Asked in the same conversation | 35.7 / 32.1 | 54.8 / 64.3 | 80.8 / 84.9 |
| Reviewing in a fresh conversation | 75.0 / 70.2 | 82.2 / 63.1 | 90.4 / 87.7 |
% rated HIGH on the attacker's call / on the matched authorised send. Only episode pairs in which both the attack and its twin produced a send are counted (84, 157 and 73 pairs), so the numbers differ slightly from Finding 2. Bold: the two cells whose separation is at least 10 points with a 95% interval above zero (+20.5 and +19.1).
Ratings only matter if they trigger a confirmation. We ran complete episodes with four harness gates and a simulated user who rejects every attacker call shown to them, which is the best case for any gate. The rule "always ask before any of five send or post functions" stopped every attack in this test, because the attacker's calls go through those functions. Its price is interruptions: it also interrupted 53–90% of the twin episodes in which the user really asked for a send. For gpt-oss-120b, the self-label gate gave no detectable reduction.
| Gate (when does the harness ask the user?) | gpt-oss-120b | Qwen3-32B | Llama-3.3-70B |
|---|---|---|---|
| No gate (never asks) | 26.5 / – | 43.0 / – | 26.1 / – |
| Self-label: the model's own label is HIGH or missing | 26.5 / 26.5 | 23.9 / 47.6 | 19.9 / 11.9 |
| Reviewer: a fresh copy of the model rates the call HIGH | 7.7 / 65.9 | 4.5 / 61.2 | 7.5 / 49.2 |
| Rule: the call uses one of five send or post functions | 0.0 / 78.2 | 0.0 / 90.0 | 0.0 / 52.6 |
Attack success rate (%) / authorised-twin episodes interrupted (%). About 715 complete episodes per model and gate, same random seeds for every gate. Attack success without a gate is lower here than in Finding 1 because these runs deliver tool outputs in the native tool role and use a different task mix.
The same weights do not make the same agent. For Llama-3.3-70B a chat role works like a badge of authority: text labelled user gets obeyed far more often than the same text labelled tool. gpt-oss-120b shows no such effect in the same setup. And in all three models, the label a model writes about its own action while acting understates what the same model flags when it steps back and reviews that action.
tool role, not as user messages.important_instructions template shown above.The repository holds the code for every experiment and compact records of every episode behind the tables above. One script recomputes the headline numbers from those records on a laptop, no GPU needed.
git clone https://github.com/EvenEureka/RoleCall.git
cd RoleCall
pip install -r requirements.txt
python reproduce/headline_numbers.py
Rerunning the agents themselves needs GPUs to serve the 32B–120B models; see the README.
These notes are numbered by finding. None of them changes the direction of a result; they say where a result holds and what it does not show.
ignore_previous and +15.3 under a bare direct template. So "no effect" holds for this attack frame only.