self-hosted-llm-batch-inference

Installation
SKILL.md

Self-Hosted LLM Batch Inference

Build a repeatable offline inference path for a startup that needs GPU economics without operating an ML platform. Use one AWS Batch job for one corpus run, backed by Amazon ECS Managed Instances Spot. Stage model weights in Amazon S3, divide the corpus into deterministic shards, and process those shards sequentially with vLLM on one GPU.

Keep the infrastructure stack separate from each run. Provision reusable Batch, ECR, IAM, logging, and optional networking resources with infrastructure as code. Submit each run separately with immutable image, model, prompt, input manifest, and output identifiers.

Set scaleInAfter to 0 for this architecture. No follow-up task is expected on the instance after the Batch job ends, so retaining idle GPU capacity adds cost without improving the run.

Scope

Apply this skill only when all of these conditions hold:

Installs
32
GitHub Stars
3.5K
First Seen
3 days ago