evaluating-llms
Warn
Audited by Socket on Sep 17, 2026
1 alert found:
SecuritySecurityexamples/python/benchmark_testing.py
MEDIUMSecurityMEDIUM
examples/python/benchmark_testing.py
The code is intended as an LLM benchmarking utility and shows no clear malware or covert backdoor behavior. However, run_code_benchmark() executes untrusted model-generated Python with exec(), creating a high-impact arbitrary code execution risk. Benchmark data is also sent to the OpenAI API by design. The supplied module additionally contains syntax errors in create_sample_code_benchmark(), preventing execution as written. Generated code should be validated and run in a strongly isolated sandbox, or execution should be removed.
Confidence: 99%Severity: 86%
Audit Metadata