CommCP: Decentralized Multi-Agent Coordination with Conformal-Prediction-Calibrated LLM Communication

This skill enables Claude to build multi-agent coordination systems where agents communicate through LLM-generated messages that are statistically calibrated using conformal prediction. Instead of agents flooding each other with every observation, each message passes through a confidence filter derived from split conformal prediction, guaranteeing that only messages exceeding a calibrated relevance threshold are transmitted. This eliminates the core problem in multi-agent LLM systems: receiver distraction from irrelevant or misleading inter-agent messages.

When to Use

• When the user asks to build a multi-agent system where agents must share observations or findings with each other and noise/redundancy is a concern
• When designing a decentralized coordination protocol where no central controller exists and agents must decide independently what to communicate
• When the user wants to filter or calibrate LLM outputs before passing them to downstream agents or consumers, using statistical guarantees rather than ad-hoc thresholds
• When implementing cooperative task allocation among heterogeneous agents that each have different capabilities or explore different regions
• When building a multi-agent question-answering or information-gathering pipeline where agents explore, observe, and share relevant findings
• When the user needs to add uncertainty quantification to LLM-generated classifications that drive inter-agent decisions

Key Technique

The Core Problem: In multi-agent systems using LLMs for communication, agents generate messages about their observations (e.g., "I found object X, it may be relevant to your task"). Raw LLM outputs are uncalibrated — a model saying "80% confident" does not mean it is correct 80% of the time. Sending all messages creates noise; naive thresholding misses important information unpredictably.

Conformal Prediction as the Solution: CommCP applies split conformal prediction to calibrate LLM-generated relevance assessments. During a calibration phase, the system collects LLM confidence scores on labeled examples (where ground truth relevance is known). It computes the quantile threshold q̂ at level (1 - ε) from these calibration scores, where ε controls the acceptable miscoverage rate (e.g., ε = 0.05 for 95% coverage). At runtime, an agent only transmits a message if the LLM's confidence score for that message exceeds q̂. This provides a distribution-free statistical guarantee: the probability that a truly relevant observation is filtered out is at most ε.