openrlhf-training
Fail
Audited by Gen Agent Trust Hub on Jun 29, 2026
Risk Level: HIGHCOMMAND_EXECUTIONREMOTE_CODE_EXECUTIONPROMPT_INJECTION
Full Analysis
- [COMMAND_EXECUTION]: The installation instructions in
SKILL.mdsuggest running Docker with the--cap-add=SYS_ADMINflag. This grants the container significant privileges on the host system, increasing the risk of container escape. - [COMMAND_EXECUTION]: The
SKILL.mdfile advises usingsudo pip uninstallfor resolving package conflicts. Executing package managers with root privileges is a security risk as it allows for arbitrary modifications to the host system during the uninstall process. - [COMMAND_EXECUTION]: In
references/custom-rewards.md, the example reward function usessubprocess.runto execute Python code viapytest. This pattern allows the execution of system commands based on the content of a temporary file containing model-generated text. - [REMOTE_CODE_EXECUTION]: The skill framework facilitates the loading and execution of arbitrary Python logic via parameters like
--remote_rm_urland--agent_func_path. This enables the execution of user-supplied or remotely referenced code within the training environment. - [PROMPT_INJECTION]: The skill architecture is susceptible to indirect prompt injection. Untrusted data from model outputs (
queries) and external labels are processed by reward functions and agents. If these scripts perform actions like code execution (as shown in the documentation examples), a malicious model response could lead to unintended command execution. - Ingestion points:
queries,prompts, andlabelsparameters inreward_func.pyandagent_func.pyreferenced inreferences/custom-rewards.md. - Boundary markers: None provided in the implementation templates to distinguish data from executable instructions.
- Capability inventory:
subprocess.runcalls documented inreferences/custom-rewards.md. - Sanitization: None; the provided example directly writes generated code to a file and executes it via
pytest.
Recommendations
- AI detected serious security threats
Audit Metadata