I Tried to Make an AI Agent Flaky. It Went 100 for 100.

·

7 min read

Cover Image for I Tried to Make an AI Agent Flaky. It Went 100 for 100.

I set out to show that AI agents are flaky. I built a task full of traps, ran the same agent 100 times, and waited for the results to scatter.

They didn't. All 100 runs gave the correct answer, and they got there by exactly the same route.

That's the opposite of what I expected from the probabilistic model, and I think it's a more useful result than the one I was going to write about.

The setup

To measure an agent, you have to hold everything else constant. So the harness fixes:

  • The prompt. It's the same string on every trial.
  • The tools. They're pure Python functions over a hardcoded dataset, with no network, no clock and no database.
  • The agent loop. It's about 40 lines with no framework and no retries. If the model gets it wrong, that counts.
  • The answer check. The correct answer is computed in code from the same data the tools serve.

The task is a small operations question:

Customer C-42 is asking for a credit on orders that contained a discontinued product. Find every order for C-42 that includes at least one discontinued product, excluding cancelled orders, and report the total value of those orders.

The agent gets three tools:

toolreturns
list_orders(customer_id, page)order IDs, 5 per page, plus next_page when there are more
get_order(order_id)status and line items
get_product(sku)unit price and active / discontinued status

The correct answer is $694.00.

The traps

I built the data so each common agent mistake would land on its own wrong number. That way a failure would tell me why it failed, not just that it failed.

answerwhat the agent did
$694.00correct
$158.00read page 1 of list_orders and never fetched page 2
$871.00included the cancelled order
$296.00summed only the discontinued line items instead of the full order value
$335.00 / $473.00combinations of the above

Customer C-42 has seven orders, so the list paginates, and two of the qualifying orders are on page 2. One cancelled order also contains a discontinued product, so it's a tempting false positive. And "orders that contained a discontinued product" can pull a model toward totaling only those lines.

The run

I ran Claude Haiku 5.5, Anthropic's smallest current model, 100 times at the API's default temperature (1.0). That is full sampling, not the near-deterministic temperature 0.

metricresult
correct answers100 / 100
distinct final answers1
distinct tool-call sequences1
tool calls per run16 every time (2 list, 7 order, 7 product)
model turns per run5 every time
distinct final replies (wording)100
output tokens per run1,431 to 1,678
wall-clock per run6.1 to 7.2 seconds

None of the traps caught it. It paged through both pages, fetched all seven orders in parallel, then all seven products in parallel, skipped the cancelled order and summed full order values. It made the same 16 calls in the same order, 100 times.

The randomness was real, but it all went into the prose. Every final reply was worded differently and varied by about 250 tokens in length. None of that variation reached a decision.

Why it didn't break

Looking back at the task, every decision point was forced:

  • Pagination announced itself. The tool returned next_page: 2. An agent doesn't have to guess whether there's more data when the response tells it.
  • Every status was binary. An order was cancelled or it wasn't. A product was discontinued or it wasn't. There was nothing to interpret.
  • The prompt closed the loophole. It says to report "the full order value, all line items." That removed the one ambiguity I'd built in.
  • The tools never misbehaved. There were no timeouts, partial responses or schema surprises.

Sampling randomness only matters at a fork in the road, where more than one next step is plausible. This task had no forks. So a model sampling at full temperature still walked a single path.

That's the real lesson for me. When agents are flaky in production, the cause usually isn't the sampler. It's ambiguity: a vague instruction, a tool response that can be read two ways, a status field with five values and no documentation, an API that fails one time in fifty. Remove those and even a small model is very consistent. Leave them in and no temperature setting will save you.

What 100 out of 100 actually proves

Here's the part I almost skipped. 100 correct runs out of 100 does not prove the agent is always right.

With zero failures in 100 trials, the 95% confidence interval for the true pass rate is about 96.3% to 100%. (The quick rule of thumb is the "rule of three": zero failures in n trials puts the upper bound on the failure rate at roughly 3/n.) So the data can't rule out an agent that fails once in every 27 runs.

That matters more than it sounds once the agent runs unattended. What you care about then isn't pass@k, the chance that at least one of k tries works. It's pass^k, the chance that all k runs work. The τ-bench authors popularized that metric for this reason. Take the worst case my data allows, 96.3%:

consecutive runsworst-case chance all are correct
196.3%
1068.6%
5015.2%
1002.3%

So even a perfect 100-for-100 test can't promise that a batch of 10 customer credits comes out clean more than about two times in three. To tighten that bound you need more trials.

What I'd take from this

Design out the forks. Before tuning prompts or temperature, list every point where your agent has to choose and ask whether the choice is really ambiguous. Make tools say what they mean, like next_page, explicit enums and full totals.

Put determinism in the tools, not the prompt. If an agent must never miss page 2, the most reliable fix is a tool that returns everything. Every decision you take away from the model is one it can't get wrong.

Test with n, not anecdotes. One passing demo tells you almost nothing. 100 passing runs tell you the pass rate is probably above 96%. Report the interval.

Report pass^k for anything unattended. If no human checks each output, a single-run pass rate overstates how often a whole batch comes out clean.

Expect the prose to vary. If your downstream code parses free text, it will see 100 different replies. Pin the parts you depend on to a strict format, like the ANSWER: line this harness requires.

Limits

This is one model, one task and one temperature setting. A larger, messier task, a vaguer prompt or a flaky tool would very likely produce failures, and that's exactly the point. This experiment measured what happens when the forks are gone, not how a model handles real ambiguity.

Run it yourself

The harness is about 600 lines of Python with no agent framework. It supports the Anthropic API and anything OpenAI-compatible, and it includes a free mock model for checking the plumbing. The raw results from this run are in the repo.

Repo: github.com/edtechniti/agent-flakiness

pip install -r requirements.txt
python run.py --provider anthropic --model <model-id> --n 100
python analyze.py

To find where your own agent breaks, edit flaky/task.py. Make the prompt vaguer, add an ambiguous status or make a tool fail sometimes, then run it 100 times. I'd like to see what you find.

Comments (1)

Discuss on Hashnode

100 for 100 on a single trapped task usually tells you the harness handles known edge cases well, not that it generalizes when tool latency, partial responses, or schema drift hit at the same time. Variance testing across perturbed environments is where flakiness actually shows up.