k8s-loki-troubleshooting

Installation
SKILL.md

Loki troubleshooting

Find out why Loki does not accept, keep or return logs, and propose a fix. Read first, ask before changing. Run the investigation commands freely. Run a command under an "Ask first" line only after the user approves that exact command; if your credentials do not allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user.

Clients (Promtail, Grafana Alloy, Fluent Bit, the OpenTelemetry Collector) push log lines to /loki/api/v1/push. The distributor validates each line (labels, size, age) and applies the tenant's ingestion rate; the ingester applies per-stream limits and writes chunks to storage; queriers answer LogQL queries. In monolithic mode one pod (-target=all) runs every part; in microservices mode each part is its own workload. Loki answers a refused push with 400 (invalid, not retried), 429 (rate limit, retried) or 401 (no tenant), and logs refused pushes as msg="write operation failed" details="<reason>", except 401s, which it does not log.

Installs
19
GitHub Stars
38
First Seen
Oct 26, 2026