add-benchmark

Installation
SKILL.md

Add Benchmark to NeMo-Gym

The Gym page Add a benchmark is the current command reference. When it and this skill disagree, the page wins. This skill adds the order of work and the checks a benchmark must pass before it merges (Step 7).

Determine Integration Type

Before starting, determine which type of benchmark you're adding:

Native benchmark — verification logic implemented directly in a Gym resources server:

  • Resources server implements verify() with reward logic
  • Agent server orchestrates model calls (use simple_agent for single-turn, or custom agent for multi-turn)
  • Example: code_gen, instruction_following, math_with_judge

External benchmark — wrapping a 3rd-party library that has its own orchestration:

  • Integrate at the agent server level (not resources server)
  • Agent's /run endpoint wraps the external library
  • Pre-process from Gym schema to library input, post-process back to BaseVerifyResponse
  • Reproduce publicly reported numbers with the original repo first, then reproduce again after Gym integration
  • Add the dependency in pyproject.toml and commit its uv.lock
Installs
1
Repository
nvidia-nemo/gym
GitHub Stars
1.3K
First Seen
9 days ago