LangChain Cost Tuning (Python)

Overview

An engineer shipped a new research agent Tuesday. By Friday the Anthropic bill had grown 6x while traffic grew 1.4x. The cost dashboard — wired to on_llm_end — showed spend up maybe 2x. Reconciling against the provider console on Monday surfaced two compounding bugs: (1) the agent's ChatOpenAI fallback kept the default max_retries=6, so each logical call billed as up to 7 requests (P30); (2) retry middleware was registered below token accounting, so every retry fired on_llm_end twice — the aggregator summed both emissions while LangSmith deduped them by generation ID, undercounting the dashboard by ~50% against actual billed rate (P25).

The fix took an afternoon: cap retries at 2, tag retries with a stable request_id, and migrate token accounting to AIMessage.usage_metadata read from astream_events(version="v2"). Finding the bug took a week. This skill is that week compressed into a runbook.