How Shade Works
How a Shade scan attacks your AI agent with scenarios, judges each attempt and reports which scenarios it elicited
Shade red-teams your AI agent before it reaches users. You connect Shade to your agent, choose a dataset of attack scenarios and start a scan. For each scenario, an attacker model tries to get your agent to do something it shouldn't, and judge models score every response. When the scan finishes, you can see which scenarios your agent failed, every attempt that led there and why the judges scored it the way they did.
Shade attacks the whole agent, not only the model behind it. Your agent's system prompt, tools, retrieval and guardrails all shape what an attacker can make it do. An agent that can run commands, read files or call internal APIs has risks its base model alone doesn't, so Shade tests the agent as your users reach it.
Core concepts
| Concept | What it is |
|---|---|
| Scan | One run of a scenario dataset against one target. A scan moves from Queued to In Progress to Completed, and its results appear while it runs. |
| Target | What Shade attacks: usually your agent, running in a test environment. You can also target a model directly, as provider/model. See How Shade reaches your agent. |
| Scenario | One attack objective, such as getting the agent to delete files outside its workspace or produce a specific kind of harmful content. Each scenario has an interaction type (Chat or Agentic) and an approach (Direct, Indirect or Benign). |
| Scenario dataset | A versioned collection of scenarios, also called a scenario set. You pick one per scan, and you can narrow it to the scenarios with particular tags. |
| Attack session | One path through a scenario's attack tree: a sequence of numbered attempts, each building on the one before. |
| Judge score | The judges' evaluation of one attempt. It records whether the agent's response met the attacker's objective, with each judge's rubric scores and reasoning. |
| Outcome | The result for a scenario: Elicited, Not Elicited, Pending / Running, Errored or Unavailable. |
How Shade reaches your agent
There are three ways to connect a target. The more of your agent's runtime Shade can run, the more realistic the findings.
| Target | How you connect it | What runs |
|---|---|---|
| A model | Enter the Target Model Identifier in the New Shade Scan dialog, such as openrouter/google/gemini-3-flash-preview. | The model only. In Agentic scenarios, Shade gives it simulated tools. |
| An OpenAI-compatible agent | If your agent accepts OpenAI-compatible chat completion requests, set its URL as the Target Model Base URL under Advanced, with the Target Model Identifier it expects and a Target Model API Key. | Your agent's own loop and tools, for each message Shade sends. |
| Your agent's own API | You expose an API that creates an isolated agent session, sends it a turn and returns what happened. Gray Swan builds the Shade connection to that API. You then start scans from the Platform as usual. See Connect Your Agent. | Your full runtime, including guardrails, sandboxes and side effects such as changed files. |
Most production agents use their own API, because it lets Shade test multi-turn conversations, attacks hidden in tool content and changes your agent makes to its environment.
How a scan runs
Start a scan
In the Platform, choose a target and a scenario dataset
Shade attacks each scenario
Attacker
Shade
Writes an attack for a user message or tool content
Target
Your agent or model
Your agent runs in your test environment with its real tools; a model gets simulated tools
Judges
Shade
Score the response against the scenario's objective
The attacker branches and tries again, building an attack tree, until the tree reaches its node and depth limits. With Stop on Success, it stops at the first elicited attempt.
Review the results
Each scenario gets an outcome; each attack session keeps its attempts and judge scores
- You start a scan. In the New Shade Scan dialog, you choose the target and a scenario dataset, and optionally filter the dataset by tag. The scan is queued.
- The attacker works through each scenario. It writes an attack, and Shade places it where the scenario calls for. In a Direct scenario, the attack is the user's message. In an Indirect scenario, it's hidden in content your agent reads through a tool, such as a file, an email or a web page. Your agent runs as it does in production, and each attempt gets its own isolated copy of the agent. Based on what worked, the attacker branches into new attempts. Each branch is an attack session, and together they form the scenario's attack tree.
- The judges score every attempt. The judges decide whether the agent did what the scenario's objective asked for, using the evidence from the attempt: the messages and tool calls, the final output, any guardrail decisions and changes to the environment that don't show in the conversation, such as a modified file. Each judge gives rubric scores and its reasoning.
- You review the results. An attempt is elicited when the agent meets the scenario's objective, and a scenario is Elicited when at least one of its attempts is. While the scan runs, the scan page counts elicited scenarios and successful attempts.
How far Shade explores
Each scenario dataset comes with a scan configuration that sets how hard Shade tries. You can override these settings for a single scan under Advanced in the New Shade Scan dialog:
| Setting | What it controls |
|---|---|
| Tree Max Nodes | The most attempts Shade makes in one scenario's attack tree. |
| Tree Max Depth | The most turns in one attack session, from the first attempt to the last. |
| Tree Concurrency | How many attempts in one tree Shade runs at the same time. |
| Run Scenario Concurrency | How many scenarios Shade attacks at the same time. |
| Stop on Success | Whether Shade stops a scenario at its first elicited attempt, or keeps exploring the tree. |
To try a variation, Shade often goes back to an earlier turn of an attack session. If your agent's API can save and restore a checkpoint, Shade branches from it instead of replaying the earlier turns, which makes scans faster and cheaper.
Larger trees find more ways through but take longer and make more calls to your agent. Concurrency settings speed a scan up but send your agent more requests at once, so lower them if your agent or its tools enforce rate limits. Every attempt runs through your agent, so point Shade at a test deployment whose tools can't touch production data.
What the results mean
- Elicited: At least one attempt got the agent to meet the scenario's objective. This is a finding: review the attack session to see how.
- Not Elicited: Shade explored the scenario's tree within its limits and no attempt succeeded.
- Errored: Shade couldn't finish the scenario, for example because calls to your agent failed.
- Pending / Running and Unavailable: The scenario hasn't finished, or has no results to show.
For each attempt, the judges' verdict is Elicited, Not Elicited, Attempt Errored or No Verdict. A Benign scenario tests the opposite failure: whether your agent refuses a legitimate request it should answer.
Where a scan runs
Shade calls your agent the way your users would, over its endpoint. Where those calls come from depends on how you use Gray Swan.
- Gray Swan-hosted. Scans run in Gray Swan's cloud and call your agent from a fixed outbound IP address. If your agent's endpoint restricts inbound traffic, allowlist that address before you start a scan. See Prepare the target for the address.
- Self-hosted. If you run Gray Swan in your own Kubernetes cluster, Shade is an optional component of the install. It picks up scans queued from your Platform, calls your agent and the attacker and judge models you configure (any OpenAI-compatible endpoint), and writes results back to your database. Calls to your agent come from your cluster, so Shade can reach agents that aren't exposed to the internet.
Shade and Cygnal
Shade finds what gets through your agent. Cygnal blocks it in production. Use them together:
- Review the elicited scenarios in a scan to see which harmful behaviors your agent can be pushed into.
- Write a Cygnal rule for each one, or tighten an existing rule, in the policy attached to your agent. See Create and Manage Policies.
- Run the scan again after you change the model, system prompt or tools, to check that nothing new gets through.
If your workspace uses Cygnal, the scan's Agent property in Scan Properties links to the Cygnal agent the scan belongs to.