long-context

Pass

Audited by Gen Agent Trust Hub on Sep 17, 2026

Risk Level: SAFEREMOTE_CODE_EXECUTIONEXTERNAL_DOWNLOADSCOMMAND_EXECUTIONINDIRECT_PROMPT_INJECTION
Full Analysis
  • [REMOTE_CODE_EXECUTION]: The skill provides instructions to clone a third-party GitHub repository (https://github.com/jquesnelle/yarn) and execute its training script (python scripts/train.py). While this is the implementation associated with the cited research paper, it involves executing code from an unverified external source.
  • [EXTERNAL_DOWNLOADS]: The skill references several external resources, including the yarn repository on GitHub and various datasets from HuggingFace (e.g., pg19, narrativeqa) for training and evaluation.
  • [COMMAND_EXECUTION]: The documentation includes multiple shell commands for setting up the environment, such as pip install transformers torch einops rotary-embedding-torch flash-attn.
  • [INDIRECT_PROMPT_INJECTION]: The evaluation code in references/fine_tuning.md demonstrates a pattern for testing long-context models that is vulnerable to indirect prompt injection. Untrusted data from external datasets is inserted directly into LLM prompts without sanitization or protective delimiters.
  • Ingestion points: External datasets (e.g., narrativeqa, pg19) loaded via the datasets library are used to provide content for prompts in the test_long_qa function.
  • Boundary markers: Absent; the prompt template f"Document:\n{document}\n\nQuestion: {question}\n\nAnswer:" lacks delimiters to segregate the untrusted document content from instructions.
  • Capability inventory: The code snippets utilize model.generate for text generation based on the resulting prompt.
  • Sanitization: The scripts do not perform any validation or escaping of the content retrieved from external sources before interpolation.
Audit Metadata
Risk Level
SAFE
Analyzed
Sep 17, 2026, 07:54 PM
Security Audit — agent-trust-hub — long-context