Why AI Workloads Queue Beside Apparently Idle GPUs
GPU utilisation usually measures the share of streaming multiprocessors active during a sampling period. It does not show whether a device is allocated or whether its VRAM can accept another workload. A model serving occasional requests may keep compute activity at 5%, while its GPU and memory remain reserved. Kubernetes normally assigns an entire GPU to one pod, so another workload waits even when the device appears mostly idle.