Test a CrewAI Project

How to test crews and flows deterministically, without API keys, and how to debug real runs with events and traces.

Verified against crewai 1.15.22 and 1.15.23 on 2026-10-01. Live-tested with real LLMs on 2026-10-01. Run crewai version first; if the major/minor differs, re-verify the version-sensitive rows with the ask-docs skill.

1. Why tests need a stub LLM

A real LLM gives different text on every run, costs money, and needs a key. Every Agent without an explicit llm resolves to gpt-4.1-mini and fails at kickoff with ValueError: OPENAI_API_KEY is required when no key is set, so a CI run with no secrets cannot even start. You cannot assert on output you cannot predict.

A stub LLM fixes the responses, so assertions are exact. It keeps every real crewai code path: prompt building, {placeholder} interpolation, task context, tool execution, guardrail retries, pydantic conversion and flow routing. It also records every call, so you can assert on the prompts crewai built. That is where most bugs show up: a missing input, a dropped context, a tool that was never listed.

Test in tiers: