Batch API Orchestrator

Category: Engineering Domain: AI Engineering

Overview

Decide when to run LLM work through an asynchronous batch API versus realtime/streaming, then design the job so it is cheap, idempotent, and resilient to partial failure. Batch APIs typically cost roughly half of realtime in exchange for higher latency (results arrive over minutes to hours, not milliseconds), which makes them ideal for evals, backfills, embeddings, and bulk classification/extraction — and wrong for anything a human is waiting on. This skill is model- and vendor-agnostic: it reasons about the batch pattern, not any one provider's API.

Clarify First

Before recommending or designing a batch job, confirm these inputs. If any is unknown or vague, ASK — do not assume:

• Latency tolerance — is a human waiting (interactive), or can results land in minutes/hours? (sets --latency-tolerance and the batch-vs-realtime verdict)
• Volume & token shape — how many requests, and the average input/output tokens each? (sets --requests, --avg-input-tokens, --avg-output-tokens for the cost estimate)
• Pricing & discount — your realtime per-token prices and the batch discount your vendor offers (sets --realtime-input-price, --realtime-output-price, --batch-discount; defaults are neutral placeholders, not real prices)

Stop rule: ask only the 2-3 that most change the output. If the user says "just draft it," proceed and list your assumptions.