I Tried to Make an AI Agent Flaky. It Went 100 for 100.
7 min read

I set out to show that AI agents are flaky. I built a task full of traps, ran the same agent 100 times, and waited for the results to scatter.
They didn't. All 100 runs gave the correct answer, and they got there by exactly the same route.
That's the opposite of what I expected from the probabilistic model, and I think it's a more useful result than the one I was going to write about.
The setup
To measure an agent, you have to hold everything else constant. So the harness fixes:
- The prompt. It's the same string on every trial.
- The tools. They're pure Python functions over a hardcoded dataset, with no network, no clock and no database.
- The agent loop. It's about 40 lines with no framework and no retries. If the model gets it wrong, that counts.
- The answer check. The correct answer is computed in code from the same data the tools serve.
The task is a small operations question:
Customer C-42 is asking for a credit on orders that contained a discontinued product. Find every order for C-42 that includes at least one discontinued product, excluding cancelled orders, and report the total value of those orders.
The agent gets three tools:
| tool | returns |
list_orders(customer_id, page) | order IDs, 5 per page, plus next_page when there are more |
get_order(order_id) | status and line items |
get_product(sku) | unit price and active / discontinued status |
The correct answer is $694.00.
The traps
I built the data so each common agent mistake would land on its own wrong number. That way a failure would tell me why it failed, not just that it failed.
| answer | what the agent did |
| $694.00 | correct |
| $158.00 | read page 1 of list_orders and never fetched page 2 |
| $871.00 | included the cancelled order |
| $296.00 | summed only the discontinued line items instead of the full order value |
| $335.00 / $473.00 | combinations of the above |
Customer C-42 has seven orders, so the list paginates, and two of the qualifying orders are on page 2. One cancelled order also contains a discontinued product, so it's a tempting false positive. And "orders that contained a discontinued product" can pull a model toward totaling only those lines.
The run
I ran Claude Haiku 5.5, Anthropic's smallest current model, 100 times at the API's default temperature (1.0). That is full sampling, not the near-deterministic temperature 0.
| metric | result |
| correct answers | 100 / 100 |
| distinct final answers | 1 |
| distinct tool-call sequences | 1 |
| tool calls per run | 16 every time (2 list, 7 order, 7 product) |
| model turns per run | 5 every time |
| distinct final replies (wording) | 100 |
| output tokens per run | 1,431 to 1,678 |
| wall-clock per run | 6.1 to 7.2 seconds |
None of the traps caught it. It paged through both pages, fetched all seven orders in parallel, then all seven products in parallel, skipped the cancelled order and summed full order values. It made the same 16 calls in the same order, 100 times.
The randomness was real, but it all went into the prose. Every final reply was worded differently and varied by about 250 tokens in length. None of that variation reached a decision.
Why it didn't break
Looking back at the task, every decision point was forced:
- Pagination announced itself. The tool returned
next_page: 2. An agent doesn't have to guess whether there's more data when the response tells it. - Every status was binary. An order was cancelled or it wasn't. A product was discontinued or it wasn't. There was nothing to interpret.
- The prompt closed the loophole. It says to report "the full order value, all line items." That removed the one ambiguity I'd built in.
- The tools never misbehaved. There were no timeouts, partial responses or schema surprises.
Sampling randomness only matters at a fork in the road, where more than one next step is plausible. This task had no forks. So a model sampling at full temperature still walked a single path.
That's the real lesson for me. When agents are flaky in production, the cause usually isn't the sampler. It's ambiguity: a vague instruction, a tool response that can be read two ways, a status field with five values and no documentation, an API that fails one time in fifty. Remove those and even a small model is very consistent. Leave them in and no temperature setting will save you.
What 100 out of 100 actually proves
Here's the part I almost skipped. 100 correct runs out of 100 does not prove the agent is always right.
With zero failures in 100 trials, the 95% confidence interval for the true pass rate is about 96.3% to 100%. (The quick rule of thumb is the "rule of three": zero failures in n trials puts the upper bound on the failure rate at roughly 3/n.) So the data can't rule out an agent that fails once in every 27 runs.
That matters more than it sounds once the agent runs unattended. What you care about then isn't pass@k, the chance that at least one of k tries works. It's pass^k, the chance that all k runs work. The τ-bench authors popularized that metric for this reason. Take the worst case my data allows, 96.3%:
| consecutive runs | worst-case chance all are correct |
| 1 | 96.3% |
| 10 | 68.6% |
| 50 | 15.2% |
| 100 | 2.3% |
So even a perfect 100-for-100 test can't promise that a batch of 10 customer credits comes out clean more than about two times in three. To tighten that bound you need more trials.
What I'd take from this
Design out the forks. Before tuning prompts or temperature, list every point where your agent has to choose and ask whether the choice is really ambiguous. Make tools say what they mean, like next_page, explicit enums and full totals.
Put determinism in the tools, not the prompt. If an agent must never miss page 2, the most reliable fix is a tool that returns everything. Every decision you take away from the model is one it can't get wrong.
Test with n, not anecdotes. One passing demo tells you almost nothing. 100 passing runs tell you the pass rate is probably above 96%. Report the interval.
Report pass^k for anything unattended. If no human checks each output, a single-run pass rate overstates how often a whole batch comes out clean.
Expect the prose to vary. If your downstream code parses free text, it will see 100 different replies. Pin the parts you depend on to a strict format, like the ANSWER: line this harness requires.
Limits
This is one model, one task and one temperature setting. A larger, messier task, a vaguer prompt or a flaky tool would very likely produce failures, and that's exactly the point. This experiment measured what happens when the forks are gone, not how a model handles real ambiguity.
Run it yourself
The harness is about 600 lines of Python with no agent framework. It supports the Anthropic API and anything OpenAI-compatible, and it includes a free mock model for checking the plumbing. The raw results from this run are in the repo.
Repo: github.com/edtechniti/agent-flakiness
pip install -r requirements.txt
python run.py --provider anthropic --model <model-id> --n 100
python analyze.py
To find where your own agent breaks, edit flaky/task.py. Make the prompt vaguer, add an ambiguous status or make a tool fail sometimes, then run it 100 times. I'd like to see what you find.
Comments (1)
100 for 100 on a single trapped task usually tells you the harness handles known edge cases well, not that it generalizes when tool latency, partial responses, or schema drift hit at the same time. Variance testing across perturbed environments is where flakiness actually shows up.
