There's a meeting that happens about 90 days after a firm ships its first agent built on a Large Language Model (LLM). Somebody asks whether the thing is actually working. The room goes quiet, because nobody has the data to answer.
That silence is not an AI problem. It's a metasystem problem wearing an AI costume.
An agent isn't a feature you shipped. It's a new autonomous actor inside a stack that was already illegible, making decisions on its own schedule, at volume, while nobody watches. Every firm running one is paying the illegibility tax on a component that acts. The 90 day meeting is when the bill comes due.
The instrument everybody reaches for first
The reflexive answer is the telemetry stack you already own. Logs, error rates, p99 latency (the response time that 99% of requests come in under). Point it at the agent, ship a dashboard, call it observability.
That survives about as long as it takes to ask 1 real question of it. The classical stack tells you whether the agent ran. It tells you nothing about whether it ran well.
The mismatch is in the unit of measurement. Traditional software has a clean one, it either does the thing it was specced to do or it doesn't. Tests pass. Bugs file. Velocity ships.
Agents don't fit that shape. The output is probabilistic, the surface area is enormous, and the failure modes are quiet ones. Wrong tone. Missed context. A confident, well formed answer that happens to be false. None of those raise an error. Every one of them shows up as a green dashboard and an unhappy customer.
So the gap is methodological, not technical. The agent was being treated as a service, and it was getting asked service shaped questions. The right question is closer to, is this agent improving as a worker?
What the program actually contains
Once you ask the worker question, the 4 pieces fall out of it. Most teams have 1 or 2.
Telemetry. Structured event logging at the level of every prompt, every retrieval, every tool call, every model response. Not "the agent ran." Every step the agent took, inputs and outputs preserved. This is the only piece that overlaps with the classical stack, and even here the granularity is different by an order of magnitude.
An eval harness. A golden dataset of inputs paired with ideal or merely acceptable outputs, run against every meaningful change. This is what catches regressions before they ship and quantifies improvements that would otherwise be invisible. Without it, "the new prompt feels better" is the entire quality process.
Cost, latency, and quality on 1 screen. Per agent, per task, over time. Cost per resolved ticket. Median time to first token. Win rate against a baseline. The dashboard isn't the point. The conversation it forces is the point, because those 3 numbers trade against each other and somebody has to make that trade on purpose instead of by accident.
A human review loop. A labeling workflow where people grade the outputs the harness can't. The labels feed back into the golden dataset. The dataset gets sharper. The harness gets more useful. The loop closes, and the closing is the whole trick, because an eval harness that never absorbs new judgment quietly decays into a regression test for last quarter's problems.
The piece that gets cut
The 4th one is the one teams drop, and they drop it for a reason that makes sense in the moment. It's the only piece that costs somebody's afternoon every week, forever. The other 3 are build once and run.
Cut it anyway and you get a program that looks complete on an architecture diagram and goes stale in a quarter. The golden dataset was assembled out of the failure modes that were known in month 1. The failure modes that actually cost you money show up in month 7, in traffic nobody anticipated, and the only mechanism that would have caught them was a person reading outputs.
Why it compounds
This matters for the same reason the metasystem matters. It compounds.
A team running the full loop ships better agents quarter over quarter, and that improvement doesn't depend on heroics. It's a property of the system rather than of whoever happens to be paying attention that month. A team without the loop ships an agent, watches it drift, ships a different agent, and calls the churn progress.
None of this is exciting work. It's instrumentation, dashboards, a weekly review, slow incremental gains. It looks exactly like the boring engineering practice it is. But 18 months out, the gap between the teams that built it and the teams that didn't is going to be the entire difference between a portfolio of assets and a graveyard of demos.
That's the part worth building now, while you still only have 1 agent to instrument.