fit-benchmark

fit-benchmark answers one question Platform Builders care about: did our skills (the fit-* / kata-* packs) make agents better at writing code? A single agent run is a coin flip; one passing eval doesn't prove anything. fit-benchmark runs each coding task N times across a skill-set manifest, grades each run against tests that never enter the agent's working directory, and reports pass@k using the OpenAI HumanEval unbiased estimator.

Why a Separate Tool

fit-eval is the generic agent-evaluation plumbing; fit-benchmark is the opinionated layer on top — task-family format, hidden invariant checks, post-hoc judge, and multi-run aggregation.

Task Family Format