agent-host-e2e-tests

Installation
SKILL.md

Agent host end-to-end tests

These tests run the whole agent host end-to-end (real server, real bundled provider SDK/CLI, real AHP protocol) while replaying recorded model traffic from committed YAML fixtures — deterministic and tokenless.

Before doing anything, read the architecture + troubleshooting reference: src/vs/platform/agentHost/test/node/e2e/README.md

It documents the mental model, the fixture format, every config flag, and a symptom→cause→fix troubleshooting table. This skill is only the workflows; the README is the source of truth for how it works.

Non-negotiable rules

  1. Replay is default and strict. No env var → serves committed fixtures, no token, no network. An unrecorded request is a hard cache miss that fails the run.
  2. A model-backed fixture's filename is derived from the test title (${provider}-${slug}.yaml). Renaming such a test orphans its fixture — re-record after any rename. Tests explicitly registered with hostOnlyTest(...) share captures/empty.yaml.
  3. Recording needs a real token (GITHUB_TOKEN or gh auth token) and talks to real CAPI. Only run it intentionally, with trivial/read-only prompts in temp dirs.
  4. Never hand-write or hand-edit fixture contents (especially not secrets/paths). Fixtures are always produced by recording; normalization/redaction is the proxy's job.
  5. Gate, don't fight. If a behavior can't replay deterministically, gate the test (see Workflow C) instead of loosening timeouts or the strict check.
  6. Track every disabled variant. Keep e2e/KNOWN_ISSUES.md current with the test title, scope, expected and observed behavior, and a focused reproduction command. Record symptoms, not speculative root causes.

Workflow A — Add a cross-provider test

Installs
9
GitHub Stars
188.3K
First Seen
Jul 10, 2026
agent-host-e2e-tests — microsoft/vscode