/ratel-langsmith-analyze — read live runs, propose fixes

Pull aggregates and outlier runs from the LangSmith MCP server (or the langsmith SDK as a fallback), pattern-match against the catalog of known agent failure modes, and write a findings report the customer can act on this week. Two grouped outputs: Ratel-flavored opportunities (where we'd integrate or deepen Ratel) and general low-hanging fruit (anyone could fix it).

The general findings are not filler — they're how we earn trust. A consultant who only ever recommends their own product looks like a salesperson. We're not that.

LangSmith's data model is projects → traces → run trees. A trace is a root run with a tree of child runs; each run has a run_type of chain, llm, tool, retriever, parser, prompt, or embedding. Sessions/conversations are carried as a thread_id / session_id metadata key set on the runs. Keep this model in mind: a "trace" in the Langfuse sense maps to a root run here, and an "observation" maps to a child run.

What good output looks like

A finding is good if:

It cites at least one run id or trace id, or a saved filter URL / filter string, so the customer can verify it themselves.
It says what to do, not just what's wrong. Vague findings ("error rate is high") waste partner time.
It says why the fix matters — in one sentence the customer's PM can read.
It's tagged Ratel or generic. Mixing them hides the value story.
If it's a Ratel-flavored finding, it cites the Ratel version that solves it — pull the conceptual signal → version mapping from the shared ratel-value-map.md (vendor-neutral source of truth; today's baseline is the v0.1.6 line).