OpenRouter Load Balancing

Overview

A single OpenRouter API key has rate limits (requests/minute and tokens/minute). To scale beyond those limits, distribute requests across multiple keys. OpenRouter also provides server-side load balancing via provider routing and the :nitro variant for low-latency inference. This skill covers multi-key rotation, health-based routing, circuit breakers, and concurrent request patterns.

Prerequisites

• Two or more OpenRouter API keys exported as OPENROUTER_KEY_1, OPENROUTER_KEY_2, OPENROUTER_KEY_3 so the KeyPool has keys to rotate — see the openrouter-install-auth skill for creating and exporting keys
• OPENROUTER_API_KEY exported for the single-key concurrent-processing pattern
• Python 3.8+ with the OpenAI SDK and requests (pip install openai requests) — the concurrent example uses AsyncOpenAI from the same package
• Adequate credits on every key in the pool; per-key quota is visible via GET /api/v1/auth/key

Instructions

1. Export your pool keys and build the KeyPool from Multi-Key Round Robin — it round-robins across keys, trips a circuit breaker after 3 consecutive errors, and auto-recovers a key after a 60s cooldown.
2. Send traffic through balanced_completion(): on RateLimitError it calls pool.mark_error(key) and retries with the next healthy key.
3. For batch workloads, use parallel_completions() from Concurrent Request Processing — an asyncio.Semaphore (max_concurrent=3-5) caps in-flight requests against a single key.
4. Layer on server-side distribution per Provider-Level Load Balancing: pass extra_body={"provider": {"order": [...], "allow_fallbacks": True}} so OpenRouter spreads the same model across Anthropic, AWS Bedrock, and GCP Vertex.
5. Monitor quota per key with check_rate_limits() (GET /api/v1/auth/key) from Rate Limit Awareness, and when 429s hit all keys simultaneously, apply the fixes in Error Handling (more keys, request queuing).