ML Inference Optimization

Overview

Use this skill for reducing inference latency, increasing throughput, shrinking memory, lowering cost, and deploying optimized models safely. Optimization must be benchmark-driven: define workload, input shapes, concurrency, SLOs, hardware, runtime, numerical tolerance, and quality metrics before changing the model.

Optimization Workflow

1. Establish a baseline with realistic data, preprocessing, postprocessing, network overhead, warmup, and concurrency.
2. Profile bottlenecks: CPU preprocessing, model compute, memory bandwidth, GPU utilization, serialization, queueing, retrieval, or downstream calls.
3. Apply the least risky optimization first: batching, compilation, precision, runtime tuning, then compression.
4. Revalidate accuracy, calibration, fairness slices, robustness, and numerical stability after every change.
5. Measure p50, p95, p99 latency, throughput, memory, cold start, error rate, and cost per 1,000 predictions.

Export and Compilation

Export paths should be chosen by deployment target. Compare exported output with framework output on representative inputs. Define tolerances per output type. Include preprocessing and postprocessing in the benchmark; many failures occur outside the core model.