What discovery provides:

datasources: All datasources with uid, name, type, isDefault (grouped by type)
dashboards: Available dashboards with uid, title, folderTitle, tags
folders: Dashboard folders with uid, title
alerts: Alert rules with total count, state breakdown, and firing alerts list
incidents: Active incidents with id, title, status
prometheus: Per-datasource label discovery (labelNames, keyLabels for job/namespace/instance)
loki: Per-datasource label discovery (labelNames, keyLabels for app/namespace/pod)
capabilities: siftAvailable - whether Sift plugin is installed (for find_error_pattern_logs)
hints.loki: Query limits, forbidden regex patterns, and safe alternatives

Why run discovery:

Get datasourceUid required for all Prometheus/Loki queries
Know available labels before building queries
Check if Sift plugin is available (determines if find_error_pattern_logs will work)
See firing alerts that need immediate attention

Smart Query Rules

// ✅ CORRECT - always specify at least one label await query_loki_logs({ datasourceUid, logql: '{app="backend"}', limit: 50 }); await query_loki_stats({ datasourceUid, logql: '{namespace="prod"}' });

Use discovery (`list_loki_label_values`) to find valid label values if you don't know them.
</critical>

| Bad ❌ | Good ✅ | Why |
|--------|---------|-----|
| `expr: 'node_cpu_total'` | `expr: 'avg(rate(node_cpu_total[5m]))'` | Aggregation reduces 100+ series to 1 |
| `queryType: "range"` for current value | `queryType: "instant"` | Single point vs 60+ points |
| `limit: 1000` | `limit: 20-50` first | Sample before fetching all |
| `startTime: "now-24h"` | `startTime: "now-15m"` | Recent data sufficient for troubleshooting |
| `query_loki_logs()` first | `find_error_pattern_logs()` first | Patterns before raw logs |

**Query progression**: Aggregated summary → Filter to problem area → Small sample → (rarely) Full data

## Tools

**Datasources:** `list_datasources(type?)`, `get_datasource(uid?, name?)`

**Dashboards:** `search_dashboards(query?)`, `get_dashboard_by_uid(uid)`, `get_dashboard_summary(uid)`, `get_dashboard_panel_queries(uid, panelId)`, `get_dashboard_property(uid, jsonPath)`, `update_dashboard(dashboard?, uid?, operations?, folderUid?, message?, overwrite?)`, `run_panel_query(dashboardUid, panelIds, queryIndex?, start?, end?, variables?)`, `search_folders(query?)`, `create_folder(title, uid?, parentUid?)`

**Prometheus:** `query_prometheus(datasourceUid, expr, startTime, queryType?, endTime?, stepSeconds?)`, `query_prometheus_histogram(datasourceUid, metric, percentile, labels?, rateInterval?, startTime?, endTime?, stepSeconds?)`, `list_prometheus_metric_names(datasourceUid)`, `list_prometheus_label_names(datasourceUid, match?)`, `list_prometheus_label_values(datasourceUid, labelName, match?)`, `list_prometheus_metric_metadata(datasourceUid)`

**Loki:** `query_loki_logs(datasourceUid, logql, startRfc3339?, endRfc3339?, limit?, direction?)`, `query_loki_stats(datasourceUid, logql, startRfc3339?, endRfc3339?)`, `list_loki_label_names(datasourceUid, startRfc3339?, endRfc3339?)`, `list_loki_label_values(datasourceUid, labelName, startRfc3339?, endRfc3339?)`, `find_error_pattern_logs(name, labels, start?, end?)`, `find_slow_requests(name, labels, start?, end?)`, `query_loki_patterns(datasourceUid, logql, startRfc3339?, endRfc3339?, step?)`

**Alerting:** `alerting_manage_rules(operation: "list"|"get"|"create"|"update"|"delete", rule_uid?, ...)`, `list_alert_groups(...)`, `alerting_manage_routing(operation: "get_notification_policies"|"get_contact_points"|..., ...)`

**Annotations:** `get_annotations(...)`, `create_annotation(...)`, `update_annotation(...)`, `get_annotation_tags(...)`

**Incidents:** `list_incidents(...)`, `get_incident(incidentId)`, `create_incident(...)`, `add_activity_to_incident(incidentId, body, eventTime?)`, `list_oncall_schedules(...)`, `get_current_oncall_users(scheduleId)`

**Search:** `search_logs(datasourceUid, pattern, table?, start?, end?, limit?)`

**Elasticsearch:** `query_elasticsearch(datasourceUid, query, start?, end?, limit?)`

**CloudWatch:** `query_cloudwatch(datasourceUid, namespace, metricName, dimensions?, statistic?, period?, start?, end?, region, accountId?)`

**ClickHouse:** `query_clickhouse(datasourceUid, query, start?, end?, variables?, limit?)`, `list_clickhouse_tables(datasourceUid)`

**Pyroscope:** `list_pyroscope_label_names(data_source_uid, ...)`, `list_pyroscope_label_values(...)`, `list_pyroscope_profile_types(data_source_uid)`, `fetch_pyroscope_profile(...)`

**Rendering:** `get_panel_image(dashboardUid, panelId?, width?, height?, timeRange?, variables?, theme?, scale?)`

**Navigation:** `generate_deeplink(resourceType, dashboardUid?, datasourceUid?, panelId?, queryParams?, timeRange?)`

**Admin:** `list_teams(query?)`, `list_users_by_org()`, `list_all_roles(delegatableOnly?)`, `get_role_details(roleUID)`

**Examples:** `get_query_examples(datasourceType)`

**Sift:** `start_sift_investigation(...)`, `get_sift_investigation_status(...)`, `list_sift_investigations(...)`

**Asserts:** `get_assertions(startTime, endTime, entityType, entityName, ...)`

## Quick Patterns

**Discover:**
```typescript
const datasources = await list_datasources({ type: "prometheus" });
const labels = await list_prometheus_label_names({ datasourceUid: "ds-uid" });
const apps = await list_loki_label_values({ datasourceUid: "ds-uid", labelName: "app" });

LogQL: {label="value"} |= "filter" - Examples: {app="backend"}, {app="backend"} |= "error", {app="backend"} | json | status >= 500

import type { QueryLokiLogsResult, QueryLokiStatsResult } from "./_skills/connections/grafana/monitoring-grafana/schemas";

// Stats query (label matchers only, no line filters)
const stats = await query_loki_stats({
  datasourceUid, logql: '{app="backend", namespace="prod"}'
}) as QueryLokiStatsResult;

// Log query with line filter
const response = await query_loki_logs({
  datasourceUid, logql: '{app="backend"} |= "error"', limit: 100, direction: "backward"
}) as QueryLokiLogsResult;
const logs = response.data;
// Strip JSON quotes bug: timestamp.replace(/^"|"$/g, '') then new Date(Number(ts) / 1e6)

// ✅ CORRECT - regex alternation '{namespace="prod"} |~ "error|ERROR|Exception|FATAL"'

// ✅ BEST - broad query + TypeScript filtering const response = await query_loki_logs({...logql: '{namespace="prod"}', limit: 100}) as QueryLokiLogsResult; const errors = response.data.filter(l => /error|exception|fatal/i.test(l.line));

</critical>

<critical>
**RATE LIMITS:** Max 2-3 concurrent queries (429 error otherwise). **Prefer single broad query + TypeScript filtering** over parallel queries.
```typescript
const response = await query_loki_logs({datasourceUid, logql: '{namespace="prod"}', limit: 100}) as QueryLokiLogsResult;
const errors = response.data.filter(l => /error/i.test(l.line)); // Filter in code

// ✅ CORRECT - label matchers only await query_loki_stats({datasourceUid, logql: '{app="backend", namespace="prod"}', ...});

// Then filter with query_loki_logs if needed const logs = await query_loki_logs({datasourceUid, logql: '{app="backend"} |= "error"', limit: 100});

</critical>

<critical>
**SIFT PLUGIN:** `find_error_pattern_logs()` / `find_slow_requests()` may return 404. Always use try-catch with manual fallback:
```typescript
try {
  patterns = await find_error_pattern_logs({name: "error-analysis", labels: {namespace: "prod"}});
} catch (e: any) {
  if (e.message?.includes('404')) {
    const patternMap = new Map<string, number>();
    logs.forEach(log => {
      const p = log.line.substring(0, 100).trim();
      patternMap.set(p, (patternMap.get(p) || 0) + 1);
    });
    patterns = Array.from(patternMap.entries()).sort((a, b) => b[1] - a[1]).slice(0, 10);
  } else throw e;
}

Best Workflow: Single broad query → TypeScript categorization → pattern detection

const response = await query_loki_logs({datasourceUid, logql: '{namespace="prod"}', limit: 100, direction: 'backward'}) as QueryLokiLogsResult;
const logs = response.data;
const errors = logs.filter(l => /error/i.test(l.line));
const patternMap = new Map<string, number>();
errors.forEach(log => {
  const p = log.line.substring(0, 100).trim();
  patternMap.set(p, (patternMap.get(p) || 0) + 1);
});
const top5 = Array.from(patternMap.entries()).sort((a, b) => b[1] - a[1]).slice(0, 5);

PromQL: avg(rate(...[5m])) by (label), topk(5, ...), histogram_quantile(0.95, ...)

// Current state (instant query)
const p95 = await query_prometheus({datasourceUid, expr: 'histogram_quantile(0.95, rate(http_duration_bucket[5m]))', startTime: "now", queryType: "instant"});
// Trend (short range)
const trend = await query_prometheus({datasourceUid, expr: 'avg(rate(http_requests[5m])) by (status)', startTime: "now-15m", queryType: "range", stepSeconds: 60});

Dashboards: search_dashboards({query}), get_dashboard_summary({uid}), get_dashboard_panel_queries({uid, panelId})

Alerts: alerting_manage_rules({operation: "list"}), alerting_manage_rules({operation: "get", rule_uid: uid})

Workflows

Errors: find_error_pattern_logs({name: "...", labels: {...}}) → query_loki_logs(limit:20) for top pattern Performance: query_prometheus(instant) P95 → if high, query_prometheus(range, now-15m) trend Alerts: alerting_manage_rules({operation: "list"}) filter state:"firing" → query_prometheus(now-15m) for alert metric

Time & Errors

Time: Prometheus: now-1h, Loki: RFC3339 2024-01-07T00:00:00Z

Common Errors:

Empty selector {} → Loki rejects logql: '{}' with parse error: unexpected }, expecting IDENTIFIER. Always include at least one label matcher (e.g., {app="backend"}, {namespace="prod"}). Use list_loki_label_values() to discover available labels.
Timestamp quotes → Strip: ts.replace(/^"|"$/g, '') then new Date(Number(ts)/1e6)
Escape sequences \w → Use [A-Za-z0-9_]
429 rate limit → Single query + code filtering
404 Plugin → Use manual fallback
query_loki_stats with filters → Stats only supports label matchers, no line filters (|=, |~, | json)

Critical Anti-Patterns

Raw metrics first: Use aggregation (sum, avg, rate, topk) before fetching series Range for current state: Use queryType: "instant" not "range" Long time windows: Start now-15m not now-1h or now-24h High log limits: Use limit: 20-50 not 500+ for exploration Logs before patterns: Use find_error_pattern_logs() or query_loki_stats() first Complex regex: Use [A-Za-z0-9_] not \w, or filter in TypeScript 3+ parallel queries: Rate limit 2-3 max, prefer single query + code filter Assuming Sift: Always try-catch with manual fallback (404 error)

Output Format

Present results as a structured report:

Monitoring Grafana Report
═════════════════════════
Resources discovered: [count]

Resource       Status    Key Metric    Issues
──────────────────────────────────────────────
[name]         [ok/warn] [value]       [findings]

Summary: [total] resources | [ok] healthy | [warn] warnings | [crit] critical
Action Items: [list of prioritized findings]

Target ≤50 lines of output. Use tables for multi-resource comparisons.

Anti-Hallucination Rules

NEVER assume resource names — always discover via CLI/API in Phase 1 before referencing in Phase 2.
NEVER fabricate metric names or dimensions — verify against the service documentation or --help output.
NEVER mix CLI commands between service versions — confirm which version/API you are targeting.
ALWAYS use the discovery → verify → analyze chain — every resource referenced must have been discovered first.
ALWAYS handle empty results gracefully — an empty response is valid data, not an error to retry.

Counter-Rationalizations

Shortcut	Counter	Why
"I'll skip discovery and check known resources"	Always run Phase 1 discovery first	Resource names change, new resources appear — assumed names cause errors
"The user only asked for a quick check"	Follow the full discovery → analysis flow	Quick checks miss critical issues; structured analysis catches silent failures
"Default configuration is probably fine"	Audit configuration explicitly	Defaults often leave logging, security, and optimization features disabled
"Metrics aren't needed for this"	Always check relevant metrics when available	API/CLI responses show current state; metrics reveal trends and intermittent issues
"I don't have access to that"	Try the command and report the actual error	Assumed permission failures prevent useful investigation; actual errors are informative

monitoring-grafana

Monitoring Grafana

Discovery