llm-council-aggregation
LLM Council Aggregation
Purpose
Combines multiple LLM outputs or judge verdicts into one final answer, score, or ranking — robustly. This is the aggregation step of a multi-model system: given N model responses or N judge verdicts on the same item(s), decide how they get turned into one result without either (a) treating naive majority vote as automatically safe, or (b) throwing away disagreement that was actually useful signal.
Core thesis: judge/model agreement is not ground truth. Frontier models share training data and alignment recipes, so their errors correlate — a panel can carry far less independent signal than its member count suggests, and judges can agree with each other while being collectively wrong relative to humans. Measure this on your own eval set before trusting any aggregation scheme; never assume it.