Overview
When an unexpected alert fires, the clock starts ticking. Most engineers instinctively turn to an AI assistant for help, but traditional assistants require you to share context about your data sources, services, connections, and relevant labels every single time. This discovery phase—described in the original material as a 'start from scratch' process—eats into precious troubleshooting minutes. Grafana Assistant eliminates this friction by building a persistent knowledge base of your infrastructure before you ask a question. It automatically studies your environment, understands what services are running, how they interconnect, which metrics matter, and where logs reside. By the time you type your first query, the assistant already has a map of your world, enabling faster, more accurate responses. This functionality is especially powerful for teams where not everyone has full infrastructure visibility—a developer can ask about upstream dependencies and get correct answers even if they have never explored those systems before.
Prerequisites
Before you can leverage Grafana Assistant's pre-learning capabilities, ensure you have the following:
- Grafana Cloud account with the Grafana Assistant feature enabled (contact your Grafana representative or check the documentation for activation).
- Connected data sources in your Grafana Cloud stack: at least one of Prometheus, Loki, or Tempo. Ideally, all three if you want the richest knowledge base.
- Basic familiarity with observability concepts (metrics, logs, traces) is helpful but not strictly required—the assistant is designed to reduce complexity.
- Permissions to view data source configurations; the assistant needs read access to discover and scan these sources.
Step-by-Step Guide: How Grafana Assistant Builds Its Knowledge Base
Grafana Assistant works entirely in the background with zero manual configuration. A swarm of AI agents performs the heavy lifting. Below are the five core steps that happen automatically.
Step 1: Automatic Data Source Discovery
The assistant first identifies all connected Prometheus, Loki, and Tempo data sources in your Grafana Cloud stack. It does not require you to specify which sources to scan; it automatically discovers them by scanning your account settings. This step is analogous to the assistant ‘seeing’ the raw materials available in your observability environment.
Step 2: Parallel Metrics Scanning
Once data sources are discovered, AI agents query your Prometheus data sources in parallel. They look for unique metric names, label combinations, and time series that represent services, deployments, and infrastructure components. For example, they might find metrics like http_requests_total with labels service=payment, deployment=prod. This scan builds a list of all observable services.
Step 3: Enrichment via Logs and Traces
The assistant then correlates Loki log streams and Tempo trace data with the metrics discovered in step 2. For each service found, it examines log formats (e.g., JSON, plaintext) and trace structures (e.g., span attributes, parent-child relationships). This correlation adds context: the assistant learns that ‘payment service’ has structured JSON logs in Loki and its traces show calls to three downstream services.
Step 4: Structured Knowledge Generation
For each discovered service group, the AI agents generate structured documentation covering five areas:
- Service identity – name, namespace, environment
- Key metrics and labels – relevant Prometheus metrics, label keys
- Deployment details – how the service is deployed (Kubernetes, containers, etc.)
- Dependencies – upstream and downstream services, based on traces and logs
- Log and trace configuration – where logs/traces are stored and their structure
This documentation is stored persistently in the assistant’s knowledge base.
Step 5: Continuous Updating
The knowledge base is not static. The assistant periodically rescans data sources (typically every few minutes) to detect changes: new services, removed components, altered metric names, or updated deployments. This ensures the assistant’s map remains current without any manual intervention.
Common Mistakes
While Grafana Assistant is designed to be hands-off, avoid these pitfalls to get the most out of it:
- Assuming it works without any data. The assistant requires at least one Prometheus, Loki, or Tempo data source connected. No data means no knowledge base.
- Forgetting to enable the assistant. Check that Grafana Assistant is activated in your account—it may not be on by default for all tiers.
- Over-relying without verification. The assistant’s knowledge is built from observability data—if your data sources misrepresent the infrastructure (e.g., missing labels, incomplete traces), the assistant’s map will be incomplete. Always cross-check critical information during incidents.
- Not updating data source configurations. If you add a new Prometheus instance or change a Loki data source name, the assistant needs time to re-scan. Ensure all sources are correctly connected and accessible.
- Ignoring permission issues. The assistant may fail to scan if the service account lacks read permissions on data sources. Double-check IAM roles if the knowledge base seems sparse.
Summary
Grafana Assistant revolutionizes incident response by pre-learning your infrastructure through automatic data source discovery, metrics scanning, log/trace enrichment, and structured knowledge generation. This persistent map eliminates the need to repeatedly share context, shaving precious minutes off troubleshooting time. With zero configuration required, it empowers every team member—even those unfamiliar with specific systems—to ask accurate questions and get immediate answers. By following the prerequisites and avoiding common mistakes, you can fully leverage this proactive observability assistant and focus on fixing issues rather than explaining your environment.