Test, trace, repeat: experimentation is becoming the engine of self-improving AI
Star Players Don’t Win Championships. Systems Do.
What the New York Knicks can teach us about AI system design
Like most basketball fans, I turned off Game 4 of the NBA Finals when the Spurs went up by 30 in the 3rd quarter, then tuned back in to watch the last minute on the edge of my seat as the Knicks pulled off an epic comeback win to go up 3-1 in the Finals. And what strikes me about the Knicks is their system: the roster construction, the rotations, the thousands of small adjustments that compound across a season. Talent gets you to the playoffs. Systems win championships.
The same thing is happening in AI right now. The launch of Anthropic’s Fable model is genuinely incredible and every frontier release raises the ceiling on what a single model can do. But that’s not the question I hear from business leaders anymore. The question I hear is: how do I make AI systems reliable, repeatable, manageable, and successful in production? Not “which star do I sign,” but “what system do I build around it.”
In this article I want to address that question through a specific lens: experimentation. As in, optimization and personalization of digital experiences like websites and mobile apps. Because experimentation is transforming from a tool that teams use into the feedback loop that agentic AI runs on.
Where Experimentation Is Today: Bottlenecked by Human Interpretation
The classic experimentation model is linear and human-gated. Hypothesis, variant, traffic, wait, interpret, decide, repeat. Each step requires a person, which puts a hard ceiling on velocity. Most teams run somewhere between ten and twenty experiments a quarter.
Here’s the thing: the bottleneck was never statistical power or traffic. It’s human interpretation latency. Every result sits in a dashboard until someone reads it, contextualizes it, and decides the next move. And the most valuable artifact of the whole process, the record of what was decided, why, and what happened next, lives in emails, powerpoint decks, Slack threads and meeting notes. Unstructured, unqueryable, and effectively lost to the organization.
That model is about to break, because agents don’t operate at human cadence. Kris Kimmerle frames the mismatch well: a human knowledge worker takes maybe fifty meaningful actions a day; an agent takes thousands. You cannot put a human approval gate in front of every one of them. And you can’t simply remove the gates either. Vin Vashishta has written candidly about giving an early agent control of his content operation and watching it destroy performance, because it had no infrastructure for learning what actually worked. Both failure modes point at the same missing layer.
What’s Changing: The Loop Is Closing
Two threads are converging.
The first is warehouse-native experimentation. Platforms like Optimizely Analytics, Databricks, and GrowthBook now evaluate experiments directly against the data in your warehouse, like revenue, retention, customer and lifetime value, rather than proxy metrics like clicks and dwell time. The experiment is finally connected to the outcome the business actually cares about.
The second is agent architectures. Agents now have tool use, planning, and memory. An agent can generate a hypothesis, create the variant via API, route traffic, wait for the warehouse to report the outcome, and record what happened, end to end.
Put those together and experimentation stops being a quarterly team activity and becomes a “primitive” of an agentic system, in other words, a native capabilities. Every agent decision, every piece of generated content, every personalization choice, every routing call, is fundamentally a hypothesis about what works. If the system can’t test that hypothesis against a real outcome and feed the result back, it isn’t agentic. It’s just prompting with extra steps.
That’s what I mean by primitive: not a tool teams use, but infrastructure the agent action layer runs on. And the reward connection is that now, by running personalization and experimentation workflows through agentic systems, you create the foundation of a self-learning for digital experience channels.
Decision Traces: Decisions Are Data
For this to work, the system needs a data structure most organizations don’t have yet. The team at AI Realized Now calls it a decision trace: a structured record of how context turned into action. The inputs gathered, the policies evaluated, the exceptions invoked, the approvals collected, the changes committed — and the outcome later observed.
Read that list again and notice something: an experiment is a decision trace. A hypothesis, a variant, an audience, a pre-committed success metric, and a measured result. Experimentation teams have been producing this artifact for twenty years. What’s new is treating it as first-class enterprise data — durable, queryable, and connected — rather than a screenshot in a readout deck.
This isn’t a vector database, and it isn’t chat history. It’s an architectural commitment: decisions and their lineage become part of the organization’s data model, the same way customer records did a generation ago.
The Reward Signal, Revisited
In my last post, I argued that the key constraint on AI systems isn’t model capability — it’s the reward signal. LLMs are stateless and locally optimizing. They have no persistent representation of whether they’re moving toward the goal. The signal has to come from outside the model.
Warehouse-native experimentation is that signal. Every closed experiment encodes which variant, for which audience, in which context, produced which real business outcome. When the agent learns “urgency-framed headlines for this segment lift revenue per session 23%,” that’s not a heuristic anymore. That’s a gradient — a direction the system can actually optimize along.
This isn’t a theoretical claim. The evidence has been hiding in plain sight for a decade. Microsoft’s experimentation team famously discovered that a simple change to how Bing displayed ad headlines — an idea an engineer had sitting in the backlog — was worth over $100 million a year, a result no executive predicted. Booking.com built its market position running more than 1,000 concurrent experiments, shipping every product change as a test. And in the AI era, the pattern is accelerating: Cursor grew from $4M to $2B in ARR in under two years, not because its model was better, but because every user interaction became a signal that made the next suggestion smarter. The lesson across all three is the same. The model matters. The feedback architecture around it matters more..
Governance: The Hypothesis Library, Augmented for the Age of AI
Anyone who ran a mature experimentation program a decade ago will remember the hypothesis library: a living document where teams accumulated learnings so the next test built on the last one instead of starting from zero. The discipline was always right. What’s changed is the consumer.
When the decision trace history is structured and queryable, the hypothesis library stops being a document humans consult and becomes the memory agents reason over to determine what to test next. Experiment N’s outcome shapes experiment N+1’s hypothesis, automatically. That’s experiment chaining, and it’s where the compounding happens, because every trace makes the next decision smarter.
It’s also exactly where governance becomes non-negotiable. ServiceNow’s research team documented what they call dynamics blindness: frontier models consistently fail to predict the cascading side effects of their own actions. An agent chaining experiments at machine speed without guardrails doesn’t just make one mistake, it compounds mistakes at the same rate it would compound learnings. The governance layer needs to be engineered like a control system: which metrics agents can optimize, which systems they can touch, what budget each experiment chain can consume, and when a trajectory drifting off course triggers human escalation. Continuous correction, not periodic review.
What This Means If You’re Building
Three takeaways, whether you’re building products or platforms:
Treat decision traces as data structures. Audit how your organization records decisions today. If the answer is “Slack and slide decks,” you’re discarding the most valuable training data your agents will ever have. Every experiment, every campaign decision, every exception should produce a structured, queryable record.
Define reward signals on outcomes, not proxies. Connect your experimentation layer to the warehouse, and pre-commit the success metric before anything ships. An agent optimizing click-through will find clicks. An agent optimizing retention will find customers. You get the behavior you measure.
Write governance rules for experiment chaining before you scale. The hypothesis library mindset, made executable: define what agents may test, how learnings carry forward, and where the human stays in the loop. The teams that engineer this layer first will run thousands of safe experiments a day while everyone else is still scheduling readout meetings.
Twelve to eighteen months from now, I don’t think we’ll talk about “running experiments” any more than we talk about “running queries.” It will just be how the system learns. The organizations that win won’t be the ones with the best model, they’ll be the ones whose systems capture the most learning per decision.
So here’s the test I’d apply to any agentic system, whether you’re building it or buying it: does every decision leave a trace, and does every trace make the next decision smarter? Answer yes to both, and you’re building a learning system, one that compounds into a real moat. That’s how systems win championships. 🏆💎
This is an ongoing conversation. I’d love to hear how you’re closing the loop in the comments.
I love the breakdown you did! Easy to digest!