semantic-consistency-auditor

Pass

Audited by Gen Agent Trust Hub on Sep 23, 2026

Risk Level: SAFEEXTERNAL_DOWNLOADSINDIRECT_PROMPT_INJECTIONMETADATA_POISONING
Full Analysis
  • [EXTERNAL_DOWNLOADS]: The skill performs runtime downloads of pre-trained AI models from established repositories.
  • Evidence: SKILL.md and scripts/main.py document that the tool downloads BERTScore models (400MB-1GB) and COMET models (500MB-1.5GB) from well-known sources such as Microsoft and Unbabel during initial execution.
  • Context: These downloads are standard for the machine learning functionality provided by the skill and target recognized industry providers.
  • [INDIRECT_PROMPT_INJECTION]: The skill processes AI-generated and expert clinical notes, which are untrusted data inputs.
  • Ingestion points: scripts/main.py accepts clinical text through the --ai-generated, --gold-standard, and --input-file CLI parameters.
  • Boundary markers: The skill implementation does not utilize explicit delimiters or 'ignore' instructions to prevent the agent from potentially obeying malicious commands hidden within the notes being compared.
  • Capability inventory: The skill can write evaluation output to local files and performs network read operations to fetch models.
  • Sanitization: No sanitization or filtering logic is present in scripts/main.py to handle adversarial text content.
  • [METADATA_POISONING]: The skill includes a metadata file (semantic-consistency-auditor_audit_result_v2.json) that contains self-referential safety and security claims.
  • Evidence: The file contains multiple hardcoded assertions such as "security": "PASS" and "gate": "PASS" intended to influence the evaluation of the skill.
  • Context: As per security analysis best practices, these self-attestations are treated as data to be evaluated and do not impact the independent verdict.
Audit Metadata
Risk Level
SAFE
Analyzed
Sep 23, 2026, 04:46 PM