Breaking: Drasi Turns Documentation Testing Into a Monitoring Problem With AI Agents
A small open-source team at Microsoft Azure has deployed AI-powered “synthetic users” to automatically catch documentation bugs, after a silent infrastructure update broke every tutorial for weeks.
Drasi—a CNCF sandbox project that detects data changes and triggers real-time reactions—saw all of its getting-started guides fail in late 2025 when GitHub upgraded its Dev Container environment, bumping the minimum Docker version.
“The Docker daemon connection broke, and every single tutorial stopped working,” said a spokesperson for the four-engineer team in Microsoft Azure’s Office of the CTO. “Because we relied on manual testing, we didn't immediately know the extent of the damage. Any developer trying Drasi during that window would have hit a wall.”
The Problem: Documentation Breaks Silently — and Often
Documentation failures come in two flavors, the team explained. The “curse of knowledge” means experienced developers write implicit steps that new users (or AI agents) cannot follow. “When we write ‘wait for the query to bootstrap,’ a new user doesn’t know to run drasi list query and watch for Running status,” the spokesperson noted.
“Silent drift” is even more insidious: code changes that rename a config file break the build immediately, but documentation that still references the old filename accumulates invisible errors until a confused user reports them.
Turning Testing into a Simulation Problem
The team’s solution treats tutorial testing as a monitoring problem—using an AI agent that acts as a “synthetic new user.” The agent has three critical traits:
- Naïve – It has no prior knowledge of Drasi; it knows only what is explicitly written.
- Literal – It executes every command exactly as written. If a step is missing, it fails.
- Unforgiving – It verifies every expected output. If the doc says “You should see ‘Success’,” but the CLI just returns silently, the agent flags it immediately.
The stack uses GitHub Copilot CLI inside Dev Containers to run through tutorials automatically—uncovering broken commands, mismatched outputs, and missing dependencies before any human encounters them.
Background: How Drasi Works
Drasi is an open-source project under the Cloud Native Computing Foundation (CNCF) that lets developers set up continuous queries to detect changes in databases, APIs, or event streams and trigger automated reactions. Its tutorials typically spin up sandbox environments with Docker, k3d, and sample databases. Any upstream dependency change—a deprecated flag, a bumped version, or a new default—can silently break the experience.
“We ship code faster than we can manually test tutorials,” the team acknowledged. “The incident forced a realization: with advanced AI coding assistants, documentation testing can be converted to a monitoring problem.”
What This Means for Open-Source Projects
This approach flips the traditional documentation maintenance model. Instead of relying on periodic manual checks or user bug reports, teams can continuously validate their guides using AI agents that simulate new users. The technique is especially valuable for projects with fast release cycles or complex sandbox setups.
“Any project that provides step-by-step tutorials can benefit from synthetic users,” said the spokesperson. “It’s not about replacing human testers—it’s about catching silent drift the moment it happens, not weeks later when a frustrated developer abandons your project.”
Industry observers see this as a blueprint for scaling documentation quality in open source, where maintainer time is scarce and first impressions matter most.