Your AI Analyst Will Be Worse Next Month

Anthropic got their internal data agent to 95% accuracy. One month later it was at 65%. Nothing broke and the model did not change. Schemas moved, definitions moved, and the context the agent depends on quietly stopped being true.

Every rollout guide documents how to reach reliability. Almost none documents how to keep it. This is the missing chapter.

Accuracy is a rate, not a property

95% was never a fact about the agent. It was a fact about a moment: how well one body of written-down meaning matched one state of the warehouse on one day. Change either side and the number moves without anyone touching the agent.

So the question that matters is not how accurate your agent is. It is how fast accuracy decays and how quickly you notice. A system at 85% that recovers in a day beats a system that hit 95% in April and has not been measured since.

Twenty minutes a week keeps it alive

Anthropic's full answer is to treat context as code. Roughly 90% of their data-model pull requests now ship with a matching context change, checked in CI. Change what a table means, change what the agent was told it means, in the same review.

The version that costs almost nothing and gets you most of the way is twenty minutes a week with one named owner. Work the conversations users flagged or abandoned. Read what new users asked first, because first questions expose what your documentation assumes and never says. Prune the context nobody uses. Then re-run your test questions and see whether the number moved. Twenty minutes is not enough to fix everything. It is enough to notice, and drift only does damage while nobody is looking.

The failure mode is that nobody owns it

When a rollout dies, the postmortem blames the data model. In our experience it is almost never the data model. It is that no single person's name was on the work. Everyone agrees the agent should stay accurate. Nobody's Thursday has twenty minutes in it. The answers get slightly worse, one at a time, and by the time somebody says out loud that it used to be better, the trust is gone and winning it back is a much bigger project.

Put the twenty minutes in someone's calendar on the day you go live, not on the day someone complains. It is the smallest intervention available, and it decides whether month six looks like month one. This is the half of the work we wrote the playbook around, and it is the stage everyone skips.

If this excites you, we'd love to hear from you. Get in touch.