OpenAI Realtime API — Voice-Native AI Conversations

Overview

The Realtime API is OpenAI's speech-to-speech interface: the model hears audio directly, keeps conversation state, calls tools and answers in audio, with barge-in and low first-audio latency. Current models are gpt-realtime-2.1 (reasoning, tool use) and the cheaper gpt-realtime-2.1-mini. Transports: WebRTC (browsers, mobile), WebSocket (servers) and SIP (phone calls).

The beta interface is gone. OpenAI removed the Realtime beta on 2026-05-12 and shut down the gpt-4o-realtime-preview models on 2026-05-07. The OpenAI-Beta: realtime=v1 header, modalities, flat voice / input_audio_format fields and response.audio.* events belong to that beta and no longer work. The older gpt-realtime and gpt-realtime-mini families are scheduled for removal on 2027-01-20, so start new work on gpt-realtime-2.1.

OpenAI also offers GPT-Live (wss://api.openai.com/v1/live/sessions) for full-duplex conversations with a delegated backend. It is a separate API with its own events (session.start, session.input_audio.append); this skill covers the Realtime API only.

Instructions

Choose the transport

• Browser or mobile: WebRTC. Your server holds the API key and either proxies the SDP offer to POST /v1/realtime/calls (unified interface) or mints an ephemeral key with POST /v1/realtime/client_secrets. WebRTC handles media, playback buffering and truncation for you.
• Server to server: WebSocket at wss://api.openai.com/v1/realtime?model=gpt-realtime-2.1 with Authorization: Bearer $OPENAI_API_KEY. You send base64 PCM yourself and manage playback.
• Phone: SIP trunk to sip:$PROJECT_ID@sip.api.openai.com;transport=tls (EU: sip-eu.api.openai.com).

If your app knows the end user, send a stable hashed ID in the OpenAI-Safety-Identifier header on the server-side request that creates the session or client secret.