mHC: Manifold-Constrained Hyper-Connections

Overview

mHC (Manifold-Constrained Hyper-Connections) stabilizes deep network training by constraining residual mixing matrices to be doubly stochastic. It provides:

• Stable Training: Lower gradient norm variance via doubly stochastic constraints
• Multiple Streams: Hyper-Connections with learnable mixing across residual streams
• Sinkhorn Projection: Log-space Sinkhorn-Knopp algorithm for doubly stochastic projection
• GPT Integration: Pattern for wrapping attention and MLP layers

Two components:

• HyperConnections Module: Core PyTorch module with H_res, H_pre, H_post matrices
• Sinkhorn-Knopp: Log-space projection to doubly stochastic manifold

Quick Reference