LangChain Incident Runbook

Overview

3:07am. PagerDuty: "LangChain p95 latency > 10s for 5 minutes." You open LangSmith, filter by service="triage-agent" over the last 15 minutes, and the first trace is 43 seconds long — an agent is on step 24 of 25 iterations, bouncing between the same two tools on a vague user prompt ("help me with my account"). The cost dashboard shows $400 spent in the last 10 minutes, up from a $6/hour baseline. This is P10: create_react_agent defaults to recursion_limit=25 with no cost cap; vague prompts never converge; the spend hits before GraphRecursionError surfaces. First move is not to push a code fix — it is to flip recursion_limit=5 via config reload and add a middleware token-budget cap per session, then deal with the stuck sessions.

Or: same alert, different signature. p95 is healthy at 1.8s, but p99 is 12s and spiky. The spikes correlate with instance starts in Cloud Run. P36: Python + LangChain + embedding preloads = 5–15s cold start; Cloud Run scales to zero by default, so first-request p99 is 10x p95. First move is --min-instances=1 (or a keepalive pinger), not more CPU.