Idle GPUs: 5 Ways to Stop Paying for AI Compute You Don't Use

Published by Rushikesh Dhule, LessBill · October 26, 2026 · Updated November 16, 2026

Your team finally got the GPUs. The model is training, the inference endpoint is live, and everyone is relieved. Then the bill lands, and it is bigger than the rest of your cloud spend combined.

Here is the uncomfortable part: most of those GPUs were not doing anything. Cast AI's 2026 State of Kubernetes Optimization report, which looked at roughly 23,000 clusters across AWS, GCP and Azure, found average GPU utilization of just 5%. Teams were requesting around 20 times more GPU capacity than they actually used.

That matters more for GPUs than for anything else in your stack. An idle CPU core costs cents per hour. An idle H100 on AWS on-demand costs roughly $6.90 per GPU-hour, which is about $5,000 a month if it sits there around the clock. An 8-GPU node left running for a month is a $40,000 line item.

Let's look at where the waste comes from and how to fix it.

1. Nobody is measuring real GPU usage

Most dashboards show whether a GPU node is allocated, not whether the GPU is busy. A pod can hold a whole GPU while it waits on data loading, sits in a notebook kernel, or serves three requests an hour. Kubernetes sees "in use." Your bill sees "in use." The GPU is mostly asleep.

Fix:

  • Install NVIDIA's DCGM exporter (or check nvidia-smi on VMs) and track GPU utilization and memory per workload, not just per node.
  • Put a simple weekly chart in front of the team: GPU-hours paid vs GPU-hours actually busy.
  • Flag any GPU workload averaging under 20% busy for review.

2. Inference endpoints that never scale down

An inference service sized for launch-day traffic usually keeps that size forever. Nights, weekends and quiet weeks all run at peak capacity.

Fix:

  • Autoscale on real signals such as queue depth or requests per second, using tools like KEDA, and let low-traffic or internal models scale to zero.
  • Use a node autoscaler (Karpenter on EKS, or GKE's autoscaler) so empty GPU nodes are actually removed, not just emptied.
  • Batch requests where latency allows. One busy GPU is far cheaper than four idle ones.

3. One small job, one whole GPU

By default, Kubernetes hands each pod an entire GPU. A small embedding model, a dev notebook or a test pipeline that needs a fraction of the card still gets all of it.

Fix:

  • Use NVIDIA Multi-Instance GPU (MIG) on A100/H100 cards to split one GPU into isolated slices.
  • For dev and light workloads, enable GPU time-slicing so several pods share a card.
  • Match the card to the job. Many inference workloads run fine on L4 or A10G instances at a fraction of H100 prices.

4. Training clusters that outlive the training run

Training is bursty. A team runs an intense job for two weeks, then the cluster sits idle until the next experiment. Research notebooks attached to GPUs are even worse: people close the browser tab, not the instance.

Fix:

  • Run training as jobs that create GPU nodes, use them, and release them when done, instead of long-lived clusters.
  • Set idle shutdown on notebooks (SageMaker, Vertex AI Workbench and most JupyterHub setups support this).
  • Use Spot or preemptible GPUs for training with regular checkpointing. Interruptions become a short pause, not lost work.

5. Reservations bought for a peak that never came

GPU scarcity pushed many teams to lock in capacity "just in case," through reserved instances, savings plans or capacity blocks. Commitments make sense for steady load, but a reservation you do not fill is the most expensive idle GPU of all, because you cannot turn it off.

Fix:

  • Compare committed GPU hours against measured busy hours every month.
  • Only commit to the baseline you have run consistently for at least a few months. Cover peaks with on-demand or Spot.
  • Route dev and batch workloads onto reserved capacity first, so you use what you have already paid for.

The real price: cost per useful GPU-hour

An hourly GPU price only tells you what you pay. What you get depends on utilisation: cost per useful hour = hourly price / utilisation. Using the roughly $6.90 per GPU-hour figure quoted above:

UtilisationCost per useful GPU-hour
5%$138.00
30%$23.00
60%$11.50
90%$7.67

At 5% you are effectively paying twenty times the list price for the work that gets done. Going from 5% to 30% cuts the cost of each useful hour by 83%.

Measure it. On a GPU machine, this logs utilisation and memory every 60 seconds:

nvidia-smi --query-gpu=index,utilization.gpu,memory.used,memory.total --format=csv -l 60

On Kubernetes, see how many GPUs each node has allocated (the Kubernetes GPU scheduling docs explain the resource name):

kubectl describe nodes | grep -A 8 "Allocated resources"

One caution when reading the numbers: "allocated" is not "busy". A pod can hold a whole GPU and use 3% of it, which is exactly the pattern in section 3. Always compare allocation with the utilisation that nvidia-smi reports.

Related guides: LLM API Costs for AI Agents: Where the Tokens Actually Go and Kubernetes Overprovisioning: Why Clusters Sit Idle and the Fix.

The bottom line

GPUs are the most expensive resource most startups have ever rented, and the industry is using about one in twenty of the hours it pays for. You do not need a FinOps team to fix this. Measure real utilization, let inference scale down, share cards for small jobs, shut down idle training and notebook capacity, and size commitments to what you actually use. Even moving from 5% to 30% utilization can cut a GPU bill by more than half.

How LessBill can help

LessBill.org connects to your AWS, GCP and Azure accounts and shows where your cloud and AI spend is going, including idle and oversized GPU capacity, forgotten resources, and commitments you are not using. You get clear, prioritized fixes instead of another dashboard to stare at. If your GPU bill has started to feel like a mystery, give LessBill a try.

Sources and further reading

Also drawn on: Cast AI 2026 State of Kubernetes Optimization Report (via IT Brief); TechRadar Pro; Spheron AWS H100 pricing guide (2026).

Comments

Popular posts from this blog

Stop Paying On-Demand Prices: Savings Plans, Reserved Instances and Spot Explained

7 Zombie Cloud Resources Quietly Draining Your Bill (and How to Kill Them)

When Does Azure SQL Serverless Beat Provisioned? The Maths