GPU Inference Decision Lab
EKS/vLLM measurements support admission, long-context boundaries, scheduler defaults, useful-work cost, and FP8 KV rejection.
- Platform
- AWS EKS + vLLM
- Scale signal
- Prometheus -> HPA
- Baseline
- 0 serving GPUs
Decisions
Architecture Decisions
Admission + readiness
Use bounded admission when requests can arrive before model readiness.
100% queued delivery; direct clients dropped 237-787 iterations.
Optimize readiness before treating node launch as the cold-start bottleneck.
NodeClaim and GPU node arrival were fast; image, container, and model readiness drove the 425-439s wait. Next: Capture first-successful-completion timing across the selected cold-start reports.
Long-context scheduling
Set a concurrency or admission boundary for 8192/300 traffic.
1.20 req/s still delivers 100%, but repeats 36.8s p95 queue delay.
Do not use seq caps or larger batched-token caps as the first 1.20 req/s fix.
seqs-16 hit 76.24s p95; seqs-24 hit 61.36s; batched-16384 hit 55.58s.
Keep vLLM dynamic defaults for current 512/128 steady and burst traffic.
Dynamic default kept the best delivery and token throughput.
Cost + autoscaling
Use batching for small-request economics, but gate burst SLO claims.
$0.019752/1K steady optimized; burst optimized p95 still 10.91s.
Keep active-pressure HPA testing, but do not treat target 8 as optimal.
Targets 2/4/6/8 were all underutilized. Next: Run a higher-pressure HPA sweep that reaches clear GPU utilization separation.
Quantization + hardware
Do not select FP8 KV for this current long-context path.
47.58-69.12% delivery versus 100% baseline.
Hold the FP4 architecture decision until B200 results exist.
EC2 UnfulfillableCapacity; no quantized artifact produced. Next: Rerun once B200 capacity can produce a comparable quantized artifact.
Evidence visual
Where the latency lives
Queue attribution
vLLM timing separates queue, TTFT, and decode; admission removes queue inflation while decode stays roughly unchanged.
1.10 req/s
stable repeat; p95 decode 29.32s
1.15 req/s
queueing repeat; p95 decode 29.41s
1.20 req/s
queue-dominated; p95 decode 29.43s
1.25 admission
59 unserved; p95 decode 29.39s
Server timing from May 17-18 direct and admission-capped long-context reports
Evidence archive
Experiments
The measured runs behind the decisions — each detail page keeps the run shape, commands, and result evidence.
Memory pressure
KV Cache vs Concurrency
Streaming latency
Prefill vs Decode Timing
Scheduler behavior
Batching Scheduler Tradeoffs
Traffic shape
Request Pattern Utilization
Capacity response
Autoscaling and Queueing Behavior
Cost efficiency
Cost per Useful Work
Quantization
FP4 Quantization Optimization