ai-data-curation-pretraining
Pass
Audited by Gen Agent Trust Hub on Oct 6, 2026
Risk Level: SAFEINDIRECT_PROMPT_INJECTIONCOMMAND_EXECUTIONEXTERNAL_DOWNLOADS
Full Analysis
- [INDIRECT_PROMPT_INJECTION]: The skill is designed to ingest and process large-scale untrusted data from the web (CommonCrawl), creating a potential surface for indirect prompt injection attacks.
- Ingestion points: The workflow involves reading CommonCrawl WARC files and other external datasets (
SKILL.md,references/web-curation-pipeline.md). - Boundary markers: The skill recommends inserting end-of-document tokens to prevent cross-document context contamination (
references/web-curation-pipeline.md). - Capability inventory: The skill utilizes tools for HTML extraction (
trafilatura), tokenization, and model training/evaluation (nanotron,lighteval) (SKILL.md). - Sanitization: Robust mitigations are included, such as PII regex cascades and classifiers, safety classifiers for hate speech/CSAM, and mandatory 13-gram decontamination against evaluation benchmarks (
references/web-curation-pipeline.md). - [COMMAND_EXECUTION]: The skill provides explicit shell commands for acquiring web-scale data using established cloud utilities.
- Evidence: Provides
aws s3 cpandaws s3 lscommands for interacting with the public CommonCrawl S3 bucket (references/web-curation-pipeline.md). - [EXTERNAL_DOWNLOADS]: The skill identifies and fetches configuration, datasets, and research tools from numerous well-known technology organizations and research repositories.
- Evidence: References resources from Hugging Face, AllenAI, EleutherAI, Together AI, Microsoft Research, and the European Commission (
data/sources.json,references/dataset-discovery.md). All identified sources are established and reputable within the AI research community.
Audit Metadata