AI Agent Failures: Model or Harness? How to Localize the Real Fault

Your AI agent failed again. Before you blame the model, ask one question: did the harness drop the instruction, or did the model see it and still not follow it? Same visible behavior, two completely different fixes. A paper published on July 30, 2026 answers this with a taxonomy of 41 failure modes, and it makes the repair-assignment problem concrete: the same AI agent failures can trace back to the model, the harness, the environment, or even the grader. Here is the framework I now use to localize agent failures before touching a single prompt.

Why AI Agent Failures Get Misdiagnosed

Most teams debug agents from the outside in. They see a system-level outcome, "the agent failed", and they guess at the cause. That guess is usually wrong, and there is a structural reason it is wrong: an agent failure is rarely one failure. It is a cascade.

A tool returns a bad result, the agent builds on it, the next tool call goes sideways, and the final output is garbage. The visible failure is the last step in the chain, but the repair belongs at the first step: the earliest failure from which execution never recovers. The taxonomy paper makes this the first rule. Label the initiating failure, not the downstream symptoms.

The second reason is bias. When a task fails, both humans and AI judges default to blaming the model, even when the real fault sits elsewhere. The paper documents this with a concrete case. An agent finished phase one of a task correctly, then waited for a scripted reply email that a bug in the evaluation harness never delivered. The human annotator labeled it Stale State Delivery on the external environment edge. The reasoning-agent judge called it an observation failure and blamed the model for not searching harder. Same trace, opposite verdicts. When evaluators cannot agree on where a failure came from, your debugging process will not either.

There is also a fourth repair that almost nobody considers. The same visible failure can call for model post-training, harness engineering, environment redesign, or benchmark repair. If you do not know which one, you are guessing at the most expensive part of the system.

Localizing AI Agent Failures: 41 Failure Modes, One Taxonomy

The paper's move is to change the unit of analysis. Instead of classifying failures by benchmark or by final outcome, it treats the interaction between two components as the unit, then assigns each failure mode to that edge plus a fault side. The components are the ones you actually run: the model, the owner (the human who set the task), the harness that manages context, memory, and tool access, the tools themselves, the local environment, the external environment, and the grader.

A few examples make it concrete:

• A tool call reports success but actually failed. If the tool wrapper suppressed the error, the fault is on the tool side. If the wrapper returned the error and the model ignored it, the fault is on the model side. Same edge, tool to model, different repair.
• An agent stops honoring an early instruction after a long session. If context compaction dropped the rationale, that is Context Rationale Erosion on the context edge, a harness-level fix. If the instruction was still there and the model drifted, that is a model-side fix.
• The agent calls a tool that is not in the provided schema. That is Tool Hallucination, model fault.
• A real service rate-limited you or blocked your IP. That is Service Failure on the external environment edge. No prompt change fixes a 429.

The taxonomy has one attribution rule that keeps it honest: assign fault to the model when a more capable model could have avoided or recovered from the failure under the same conditions. That rule is why most of the 41 modes land model-side. It is not bias, it is a tiebreaker.

The paper grounds all of this in real systems. The same vocabulary applies to Claude Code, Codex, OpenClaw, and Hermes Agent, which is the part that made me stop skimming: when a taxonomy paper has to cite the agent runtime you ship on, it is describing your production environment, not a toy benchmark.

Where AI Agent Failures Actually Originate: Model, Harness, or Environment

Once you know the edge and the fault side, the fix routes itself:

• Model-side failures are targets for post-training, better system prompts, or better tool descriptions. This is the layer everyone already works on.
• Harness-side failures point to scaffolding: context management, compaction policies, error propagation, retry semantics, tool schema validation. This is the layer most teams barely audit.
• Environment and grader failures mean the evaluation conditions are wrong, and no amount of agent work will fix them.

The harness layer is bigger than people think. Another 2026 paper, Adapting the Interface, Not the Model, showed you can take a frozen LLM agent and improve it meaningfully by adapting the runtime harness alone, with zero weight changes. Many of the failures that look like model stupidity are interface mismatches at the model-environment boundary.

I have written about both sides of this before. The memory side, context compaction dropping the rationale mid-task, is exactly what I covered in my post on AI agent memory failures beyond the context window. The tool side, wrappers swallowing errors so the model never sees them, is why I wrote about MCP and safe tool use. Both are harness problems wearing a model costume.

The grader layer is the one almost nobody audits, and it is quietly dangerous. A separate July 2026 audit of agent-safety benchmarks, Safety or Just Capability?, found that an "always positive" policy, one that flags every trace as unsafe, scores F1 0.690 on R-Judge, which beats five of the 21 models that actually discriminate. Three broad-coverage benchmarks ranked the same 18 models differently. If your grader is that noisy, your failure labels are noise too. I went after the same problem from the output side in my tool-call verification post: if you cannot prove the agent did the work, you cannot tell which layer broke.

A 5-Minute Triage for the Next Agent Failure

I keep this sequence in every agent repo I touch. It costs five minutes and it stops the blame-the-model reflex:

1. Reproduce and save the full trace before changing anything.
2. Walk the trace backward to the earliest failure from which execution never recovered.
3. Name the edge: which two components were interacting when it broke?
4. Apply the more-capable-model test to set the fault side.
5. Route the fix: model, harness, environment, or grader. Only touch the prompt if the test says model.

A worked example. Your agent was supposed to update a database record, and it reports the update succeeded, but the record did not change. Step 2 finds the failure at the tool call. Step 3 names the edge: tool to model. Step 4 asks the question: would a more capable model have caught this? If the wrapper returned a 200 but silently discarded the write, the answer is no, and the fault is on the tool side. You fix the wrapper, not the prompt. If the wrapper returned an explicit error and the model pressed on anyway, the fault is model-side, and you fix the prompt or the tool description.

The Honest Part

Most agent debugging is prompt roulette. Teams see a failure, rewrite the system prompt, ship it, and call it fixed. The paper's results suggest why that habit survives: the model is the only component you can edit in a meeting. Harness bugs get swallowed by wrappers, environment failures get logged as agent errors, and graders are usually invisible to the agent entirely.

So my rule inverts the default: assume harness, environment, or grader first, model second. Model work is the most expensive fix you can make, and it is the least reversible. A prompt change that papers over a harness bug means you will rediscover the bug at the worst possible moment, usually in production. The same logic applies to how you test agents. If your test suite only checks final output, it will pass a broken agent that swallowed a tool error; it will also fail a good agent whose grader is broken. That is why I push for harness-level assertions in my testing post, not just final-output checks.

The Bottom Line

Ask "model or harness?" before you ask "what should the prompt say?" The interaction-centric taxonomy gives you a repeatable way to answer it: find the earliest failure, name the two components, set the fault side with the more-capable-model test, and route the fix to the layer that actually owns it. The authors validated the categories with four frontier models acting as independent judges; the strongest judge reached Cohen's kappa of 0.76 against human labels, with 80% category accuracy, and the judges agreed with each other almost as strongly. The structure is real, not annotator preference.

That is the whole framework: 41 failure modes, one question. Model, or harness?

Paper: Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures, arXiv:2607.28802.