Duolingo ML in Production

How to tell when a learned model earns its keep against hand-written rules, what to judge it on, and how to run a generation pipeline that does not manufacture errors quietly.

Distilled from Duolingo's engineering and product blog, where models decide what each user sees, rank what staff look at, and produce content no team could author by hand. The reader here is building something else entirely — every node ends in questions about your product, not theirs.

This skill does not own LLM agent orchestration. Agent runtimes, durable workflows, graders and internal agent tooling belong to [[../duo-ai-agent-platform/SKILL]]. This one owns trained models and generation pipelines that make or produce the product.

Model versus rules — when learning wins

• [[references/replace-the-rule-tree-with-one-learned-decision]] — years of won experiments compound into logic nobody can reason about; collapse the tree into one call, and scope v1 to need no new infrastructure.
• [[references/rank-the-noise-dont-filter-it]] — when 90% of submissions are wrong, reorder them instead of discarding them, and capture the reviewer's verdict as training data.
• [[references/shrink-the-candidate-set-instead-of-speeding-the-search]] — attack the exponent with cheap heuristics, then spend all your compute on the survivors.

Features and cold start

• [[references/shallow-features-let-new-segments-inherit-performance]] — surface-only features let one pooled model serve every segment, so a week-old segment inherits everything the others learned.

Evaluation — measuring the thing users feel