ragmir-evaluate
Installation
SKILL.md
Evaluate Ragmir retrieval
Retrieval quality depends on the corpus and on the questions people ask. Measure it on the user's own questions, change one thing at a time, and keep a change only when the numbers say so.
1. Write a golden file
Collect 20 to 50 questions to start: real ones from users, tickets, or the agent's own sessions, phrased the way people ask them rather than copied from headings. Mix exact identifiers, paraphrases, questions whose answer spans several documents, and the languages people use. Add hard negatives: plausible questions the documents do not answer, including near-domain ones that share the vocabulary but miss the fact. Ragmir must return nothing for them.