ai-data-curation-pretraining

Pass

Audited by Gen Agent Trust Hub on Oct 6, 2026

Risk Level: SAFEINDIRECT_PROMPT_INJECTIONCOMMAND_EXECUTIONEXTERNAL_DOWNLOADS
Full Analysis
  • [INDIRECT_PROMPT_INJECTION]: The skill is designed to ingest and process large-scale untrusted data from the web (CommonCrawl), creating a potential surface for indirect prompt injection attacks.
  • Ingestion points: The workflow involves reading CommonCrawl WARC files and other external datasets (SKILL.md, references/web-curation-pipeline.md).
  • Boundary markers: The skill recommends inserting end-of-document tokens to prevent cross-document context contamination (references/web-curation-pipeline.md).
  • Capability inventory: The skill utilizes tools for HTML extraction (trafilatura), tokenization, and model training/evaluation (nanotron, lighteval) (SKILL.md).
  • Sanitization: Robust mitigations are included, such as PII regex cascades and classifiers, safety classifiers for hate speech/CSAM, and mandatory 13-gram decontamination against evaluation benchmarks (references/web-curation-pipeline.md).
  • [COMMAND_EXECUTION]: The skill provides explicit shell commands for acquiring web-scale data using established cloud utilities.
  • Evidence: Provides aws s3 cp and aws s3 ls commands for interacting with the public CommonCrawl S3 bucket (references/web-curation-pipeline.md).
  • [EXTERNAL_DOWNLOADS]: The skill identifies and fetches configuration, datasets, and research tools from numerous well-known technology organizations and research repositories.
  • Evidence: References resources from Hugging Face, AllenAI, EleutherAI, Together AI, Microsoft Research, and the European Commission (data/sources.json, references/dataset-discovery.md). All identified sources are established and reputable within the AI research community.
Audit Metadata
Risk Level
SAFE
Analyzed
Oct 6, 2026, 02:03 PM