Creating Harbor Benchmark Adapters

Adapters convert external benchmarks (SimpleQA, GAIA, AiderPolyglot, CodePDE, spider2-dbt, etc.) into Harbor's standardized task directory format. Each adapter reads source benchmark data and generates many individual task directories, one per benchmark instance.

When to Use an Adapter vs. Creating Tasks Directly

Use an adapter when:

• You have an existing benchmark dataset with many instances
• Tasks share the same structure but differ in data (questions, code, etc.)
• You want to track evaluation parity with the original benchmark

Create tasks directly when:

• You're authoring original evaluation challenges
• Each task has unique structure and environment
• There are fewer than ~10 tasks

Adapter Directory Structure