generate-rag-dataset
Generate a RAG Evaluation Dataset
This recipe analyzes your RAG knowledge base and generates a comprehensive Q&A evaluation dataset.
Step 1: Analyze the Knowledge Base
Read the codebase to find the knowledge base:
- Document files (PDFs, markdown, text files)
- Database schemas (if documents are stored in a DB)
- Vector store configuration (what's being embedded)
- Chunking strategy (how documents are split)
Read every document you can access. Understand:
- What topics does the knowledge base cover?
- What's the depth of information?
- What terminology is used?
- What are the boundaries (what's NOT covered)?
Step 2: Generate Diverse Question Types
More from langwatch/skills
evaluations
Set up comprehensive evaluations for your AI agent with LangWatch — experiments (batch testing), evaluators (scoring functions), datasets, online evaluation (production monitoring), and guardrails (real-time blocking). Supports both code (SDK) and platform (CLI) approaches. Use when the user wants to evaluate, test, benchmark, monitor, or safeguard their agent.
51scenarios
Test your AI agent with simulation-based scenarios. Covers writing scenario test code (Scenario SDK), creating platform scenarios via the `langwatch` CLI, and red teaming for security vulnerabilities. Auto-detects whether to use code or platform approach based on context.
50tracing
Add LangWatch tracing and observability to your code. Use for both onboarding (instrument an entire codebase) and targeted operations (add tracing to a specific function or module). Supports Python and TypeScript with all major frameworks.
46level-up
Take your AI agent to the next level with full LangWatch integration. Adds tracing, prompt versioning, evaluation experiments, and simulation tests in one go. Use when the user wants comprehensive observability, testing, and prompt management for their agent.
38prompts
Version and manage your agent's prompts with LangWatch Prompts CLI. Use for both onboarding (set up prompt versioning for an entire codebase) and targeted operations (version a specific prompt, create a new prompt version). Supports Python and TypeScript.
37analytics
Analyze your AI agent's performance using LangWatch analytics. Use when the user wants to understand costs, latency, error rates, usage trends, or debug specific traces. Works with any LangWatch-instrumented agent.
32