framework-benchmark
Codex compatibility note:
- Invoke
$skill-name: load the registered.agents/skills/<name>/SKILL.mdand execute it with Codex tools; no separateSkilltool or Claude mode switch is needed. When a workflow step or another skill names a skill, read that file in full yourself (every line, with no line limit; in parts only when output is cut off): a skill whose implicit invocation is off is never loaded for you. Author.claude/**; mirrors are generated.- Track steps before execution and keep progress/evidence current. Explicit skill requests execute their protocol. Use the native question tool for required questions.
- Map foreign tool names to available host capabilities. A name mismatch needs no approval; an actually unavailable required capability stops with its step/evidence and a user question before adaptation.
- Subagent authorization: when a skill is user-invoked or AI-detected and its protocol requires subagents, that skill activation authorizes use of the required
spawn_agentsubagent(s) for that task.- Preserve protocol order unless the user approves a deviation. Workflows use
$start-workflow: gates stay fixed; other steps may flex with logged reasons and evidence.
Framework Benchmark
Quick Summary
Goal: Measure whether the framework earns its cost on the selected tasks.
Summary: Choose the project mode, freeze inputs/checks and runtime, execute paired isolated attempts, retain their observable journeys, assess work independently, then report outcomes, resource use and limits.
Workflow: qualify → prepare and calibrate → run → independently review and validate → report → compare across tasks.
Key Rules: Freeze the same brief, application baseline, backend, CLI version, requested models and limits; compare raw execution with the complete framework treatment, then run independent checks and anonymous reviewer sessions.
Default to an available logged-in Claude Code or Codex CLI on the host. Native mode uses trusted task/code and normal CLI permissions; worktrees isolate source/context, not hostile code. Keep holdouts out of solver inputs and unknown measurements explicit. Docker remains an optional stronger execution boundary.