k8s-loki-troubleshooting
Loki troubleshooting
Find out why Loki does not accept, keep or return logs, and propose a fix. Read first, ask before changing. Run the investigation commands freely. Run a command under an "Ask first" line only after the user approves that exact command; if your credentials do not allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user.
Clients (Promtail, Grafana Alloy, Fluent Bit, the OpenTelemetry Collector) push log
lines to /loki/api/v1/push. The distributor validates each line (labels, size,
age) and applies the tenant's ingestion rate; the ingester applies per-stream limits
and writes chunks to storage; queriers answer LogQL queries. In monolithic mode one pod
(-target=all) runs every part; in microservices mode each part is its own workload.
Loki answers a refused push with 400 (invalid, not retried), 429 (rate limit,
retried) or 401 (no tenant), and logs refused pushes as msg="write operation failed" details="<reason>", except 401s, which it does not log.