Conformance for Inference: How We Reduced Bad Deploys on a GPU Platform

Abstract

Inference on GPUs fails in repetitive ways: wrong image or artifact, mismatched CUDA or runtime, undersized GPU memory, bad resource requests, or a model that passes offline checks but regresses under real traffic. On a shared Kubernetes GPU platform, those mistakes become multi-tenant incidents: noisy neighbors, OOMKills, SLO breaches, and rollbacks that waste accelerator time. This talk describes how one team built conformance for inference workloads, checks applied before production traffic, covering container and model artifacts, GPU capacity and visibility contracts, health and readiness semantics, and minimum observability through metrics and traces where used. Attendees leave with a practical checklist they can reuse: - How to separate builds from serving conformance, - How to catch regressions early, and - How to align GPU scheduling and quotas with inference SLOs. We share what worked, what did not, what teams pushed back on, and a short checklist for platform and application owners.