agent-host-e2e-tests
Installation
SKILL.md
Agent host end-to-end tests
These tests run the whole agent host end-to-end (real server, real bundled provider SDK/CLI, real AHP protocol) while replaying recorded model traffic from committed YAML fixtures — deterministic and tokenless.
Before doing anything, read the architecture + troubleshooting reference:
src/vs/platform/agentHost/test/node/e2e/README.md
It documents the mental model, the fixture format, every config flag, and a symptom→cause→fix troubleshooting table. This skill is only the workflows; the README is the source of truth for how it works.
Non-negotiable rules
- Replay is default and strict. No env var → serves committed fixtures, no token, no network. An unrecorded request is a hard cache miss that fails the run.
- A model-backed fixture's filename is derived from the test title (
${provider}-${slug}.yaml). Renaming such a test orphans its fixture — re-record after any rename. Tests explicitly registered withhostOnlyTest(...)sharecaptures/empty.yaml. - Recording needs a real token (
GITHUB_TOKENorgh auth token) and talks to real CAPI. Only run it intentionally, with trivial/read-only prompts in temp dirs. - Never hand-write or hand-edit fixture contents (especially not secrets/paths). Fixtures are always produced by recording; normalization/redaction is the proxy's job.
- Gate, don't fight. If a behavior can't replay deterministically, gate the test (see Workflow C) instead of loosening timeouts or the strict check.
- Track every disabled variant. Keep
e2e/KNOWN_ISSUES.mdcurrent with the test title, scope, expected and observed behavior, and a focused reproduction command. Record symptoms, not speculative root causes.