Cutting agent token costs 75% with an agentic eval loop.

After migrating the agent from the direct OpenAI API to the Assistants API — and shipping a wave of new user-facing features on top of it — Sporty’s token costs skyrocketed. The features were the right bet; the cost curve wasn’t. The brief was to bring it back down without losing what made the agent worth running in the first place.

~75%↓

Token cost reduction

~95%

Agent accuracy, held constant

Faster

Per-request latency, not just cheaper

The shape of the loop

The approach was an agentic eval loop: the agent itself checked out git worktrees and wrote variations on its own code — different models, different caching strategies, different combinations — and the loop graded each variant against a rubric.

The rubric was the whole game. We set up a separate grader model — the original GPT model that had produced the baseline — and used it to score agent output against the rubric across a representative eval set. That same rubric, applied to production on the most expensive model, scored ~95%. That number became the floor: any variant that didn’t hold it was out, no matter how cheap.

What we swept

• Models — swapped across providers (OpenAI, AWS Bedrock, Ollama) and across tiers within each, looking for the cheapest model that still passed the rubric on this workload.
• Caching strategies — different prompt-caching and response-caching configurations, since a lot of the new features reused overlapping context.
• Combinations — the loop ran models and caching strategies in combination, not isolation, because the cheap option on paper wasn’t always the cheap option once caching interacted with it.

What the agent did itself

The “agentic” part wasn’t a slogan. The agent checked out git worktrees and wrote variations on the code — its own prompts, routing, and caching logic — then handed each variant to the eval loop. We weren’t hand-tuning prompts and re-running the suite; the agent was generating candidates, the grader was filtering them, and the loop was converging.

Where it landed

Same agent, same ~95% rubric score, ~75% lower token cost, and faster per request — because the cheaper configurations also tended to be the ones that didn’t carry the overhead of the expensive path. The features stayed. The cost curve didn’t.