long-context
Pass
Audited by Gen Agent Trust Hub on Sep 17, 2026
Risk Level: SAFEREMOTE_CODE_EXECUTIONEXTERNAL_DOWNLOADSCOMMAND_EXECUTIONINDIRECT_PROMPT_INJECTION
Full Analysis
- [REMOTE_CODE_EXECUTION]: The skill provides instructions to clone a third-party GitHub repository (https://github.com/jquesnelle/yarn) and execute its training script (python scripts/train.py). While this is the implementation associated with the cited research paper, it involves executing code from an unverified external source.
- [EXTERNAL_DOWNLOADS]: The skill references several external resources, including the yarn repository on GitHub and various datasets from HuggingFace (e.g., pg19, narrativeqa) for training and evaluation.
- [COMMAND_EXECUTION]: The documentation includes multiple shell commands for setting up the environment, such as pip install transformers torch einops rotary-embedding-torch flash-attn.
- [INDIRECT_PROMPT_INJECTION]: The evaluation code in references/fine_tuning.md demonstrates a pattern for testing long-context models that is vulnerable to indirect prompt injection. Untrusted data from external datasets is inserted directly into LLM prompts without sanitization or protective delimiters.
- Ingestion points: External datasets (e.g., narrativeqa, pg19) loaded via the datasets library are used to provide content for prompts in the test_long_qa function.
- Boundary markers: Absent; the prompt template f"Document:\n{document}\n\nQuestion: {question}\n\nAnswer:" lacks delimiters to segregate the untrusted document content from instructions.
- Capability inventory: The code snippets utilize model.generate for text generation based on the resulting prompt.
- Sanitization: The scripts do not perform any validation or escaping of the content retrieved from external sources before interpolation.
Audit Metadata