The riskiest moment in an agentic firm's life is the cutover from "humans do the work" to "agents do the work." Shadow mode is the pattern that makes that cutover safe.
What shadow mode is
The agent runs through its full decision logic for every case — but its output doesn't reach the customer. Humans continue to handle the case normally. The agent's would-be decision is logged alongside the human's actual decision, building an eval set in real time.
After 30–90 days, the team has:
- A real-world eval set (hundreds to thousands of cases with parallel agent + human decisions).
- A measured disagreement rate.
- The actual cost per case for the agent at production volume.
- Knowledge of the failure modes that don't show up in synthetic eval sets.
When disagreement-with-human is below the target threshold (typically <5% for routine work, <2% for regulated work), the team flips the agent into HITL mode (per the HITL spoke). From HITL, autonomous mode is a smaller step.
Why this works better than synthetic evals
A synthetic eval set has known-good answers labeled by a senior practitioner. It's a useful proxy for quality. But it has three blind spots:
- Distribution drift. The eval set reflects historical cases; production traffic shifts faster.
- Edge cases. Senior practitioners write evals for the cases they thought about; the agent will encounter cases they didn't.
- Calibration. The eval measures accuracy; it doesn't measure how confident the agent is when it's right vs. wrong. Shadow mode measures both.
Real-world parallel data catches what synthetic evals miss. Both layers — synthetic + shadow — produce a more reliable cutover decision than either alone.
What the substrate handles
In the AgentsBooks substrate, every Heart task can be flagged shadow_mode: true. When enabled:
- The agent runs through its full logic.
- The action layer is suppressed (no emails sent, no records written downstream, no money moved).
- The would-be decision lands in episodic memory (per Pillar P8).
- The operator dashboard exposes a diff per case: agent decision vs. human decision.
When disagreement-with-human stabilises, an operator flips shadow_mode: false and the same task starts producing real output.
Common failure modes during shadow
Three patterns we see consistently:
- The agent over-classifies as "high risk." Initial deployments tend to be cautious; the agent escalates everything. Tuning the confidence threshold + sharpening the system prompt brings escalation rate back to baseline.
- The agent and the human use different evidence. Sometimes the agent draws on a different policy section than the human did. Reconciling this surfaces gaps in the firm's Knowledge primitive.
- The agent is faster than the human and shifts the workload. If the agent finishes a 4-hour task in 4 minutes, the operator queue floods with shadow-mode reviews. Pacing the review load matters during long shadow runs.
How long should shadow mode last?
Depends on volume + stakes:
| Volume / Stakes | Shadow duration |
|---|---|
| High volume, low stakes (customer-support tier-1) | 4–6 weeks |
| Medium volume, regulated (KYC tier-2) | 8–12 weeks |
| Low volume, high stakes (audit sign-off) | 16–24 weeks |
| Very low volume (M&A diligence) | Until enough sample size; possibly never autonomous |
The criterion for ending shadow isn't time; it's sample-size + disagreement-rate stability. If the metrics haven't stabilised, more time is needed.
FAQ
Q: What if the customer's case suffers because shadow-mode parallelism adds latency?
A: It doesn't. The agent runs in parallel with the human, not in series. Customer-facing latency is unchanged.
Q: How is this different from canary deployment in software?
A: Conceptually similar but the comparison is human-vs-agent, not old-version-vs-new-version. The eval set comes from the disagreement.
Q: How does this map to the HITL patterns?
A: Shadow is the zeroth pattern before HITL. Once shadow ends, HITL begins. Once HITL is stable, autonomous is possible.
Want shadow-mode wired into your rollout? Start free →
