LLM-Based Relevance Scoring

Overview

Use a Large Language Model as a judge to assign 0-100 relevance scores to individual content items, evaluating substance over surface-level signals. The core principle: a well-calibrated LLM with an explicit rubric scores content more consistently than ad-hoc heuristics, but scores are measurements -- not ground truth -- and require calibration, spot-checks, and documented rubrics.

When to Use

• Corpus of text items (posts, comments, articles, reviews, messages) needs quality or relevance ranking
• Need to separate substantive contributions from low-effort content (one-liners, memes, "+1" replies)
• Identifying "Authority Peaks" -- content where a user served as a primary knowledge source for others
• Building a scoring pipeline for batch processing with checkpointing
• Need reproducible, documented scoring criteria rather than subjective impressions

When NOT to use:

• Corpus has fewer than 20 items -- manual review is faster and more reliable
• Content is non-textual (images, audio) without transcription
• You need real-time scoring with sub-second latency (use heuristics instead)
• A simple keyword or regex filter would suffice (do not use an LLM for what grep can do)