agent-usability-test
Installation
SKILL.md
Give agent a goal. Make tool available. Don't mention the tool. Observe. Run WITHOUT baseline. Delta = lift — the only metric that matters. Lift is inversely proportional to model capability: docs matter most for weak models; strong models self-correct from API responses.
Failure modes
| # | Failure | Signal |
|---|---|---|
| 1 | Non-discovery | Tool never called despite being available and relevant |
| 2 | Wrong selection | Agent picks suboptimal tool when multiple are available |
| 3 | Parameter cargo-culting | Agent copies doc examples instead of adapting |
| 4 | Response-schema blindness | Correct call, wrong field extracted |
| 5 | Auth/error cliff | 401/429/timeout → agent gives up instead of recovering |