self-hosted-llm-batch-inference
Self-Hosted LLM Batch Inference
Build a repeatable offline inference path for a startup that needs GPU economics without operating an ML platform. Use one AWS Batch job for one corpus run, backed by Amazon ECS Managed Instances Spot. Stage model weights in Amazon S3, divide the corpus into deterministic shards, and process those shards sequentially with vLLM on one GPU.
Keep the infrastructure stack separate from each run. Provision reusable Batch, ECR, IAM, logging, and optional networking resources with infrastructure as code. Submit each run separately with immutable image, model, prompt, input manifest, and output identifiers.
Set scaleInAfter to 0 for this architecture. No follow-up task is expected on the instance after
the Batch job ends, so retaining idle GPU capacity adds cost without improving the run.
Scope
Apply this skill only when all of these conditions hold: