Explicit Multi-head Attention with Inter-head Interaction (MEA)

This skill enables Claude to implement Multi-head Explicit Attention (MEA), an attention variant from Peng et al. (2026) that explicitly models cross-head interaction in Transformer models. MEA introduces two components on top of standard multi-head attention: (1) a Head-level Linear Composition (HLC) module that applies learnable linear combinations to key and value vectors across heads, and (2) head-level RMSNorm that stabilizes the recombined representations. The technique improves pretraining robustness, allows larger learning rates for faster convergence, and enables a practical KV-cache compression strategy that halves memory usage with minimal accuracy loss.

When to Use

• When the user is building or modifying a Transformer model and wants attention heads to share information rather than operate independently
• When implementing a custom attention layer in PyTorch/JAX and the user asks for "inter-head interaction" or "cross-head mixing"
• When the user needs to reduce KV-cache memory during LLM inference without retraining from scratch
• When pretraining a language model and encountering instability at higher learning rates
• When the user wants to compress a multi-head attention model by replacing physical heads with fewer "virtual heads" reconstructed via linear combination
• When adapting a pretrained model (continued pretraining) and the user wants to inject MEA layers with SVD-based initialization

Key Technique

Head-level Linear Composition (HLC): Standard multi-head attention projects inputs into h independent heads that never communicate. HLC adds a learnable weight matrix W_lc ∈ R^{h' x h} that linearly recombines heads before attention computation. Concretely, given component key tensors K_comp ∈ R^{n x h' x d} (n = sequence length, h' = component heads, d = head dimension), HLC produces composite keys via the einsum: K_lc = einsum("n h' d, h' h -> n h d", K_comp, W_lc^K). The same operation is applied separately to values with its own matrix W_lc^V. This is cheap -- only 2 * h' * h additional parameters per layer -- but it allows every composite head to be an arbitrary linear mix of all component heads, enabling rich inter-head communication.

Head-level RMSNorm: After HLC recombines the heads, their statistical properties can diverge, destabilizing training. MEA applies RMSNorm across the head dimension to the concatenated output before the final projection. This normalization preserves representational diversity while preventing gradient explosion, which is why MEA tolerates learning rates up to 3x larger than standard MHA (e.g., 3e-3 vs 1e-3).