GPU Utilization Says 100% — But What Is the GPU Actually Doing?
The setup:
- GPU: NVIDIA Tesla T4
- Environment: Google Colab
- Workload: 8000×8000 matrix multiplication (PyTorch)
- Monitoring: nvidia-smi, sampled once per second I ran the experiment in three stages: 10 seconds of cold idle, 120 seconds of sustained load, and 20 seconds of cooldown. The load and the monitoring were launched together from a single cell, like this:
import subprocess, torch, time
# Start nvidia-smi as a separate OS process so it samples while the load runs.
poller = subprocess.Popen([
"nvidia-smi",
"--query-gpu=timestamp,temperature.gpu,power.draw,utilization.gpu,memory.used,clocks.sm",
"--format=csv", "-l", "1", "-f", "gpu_log.csv"
])
time.sleep(10) # cold-idle baseline
x = torch.randn(8000, 8000, device='cuda')
end = time.time() + 120 # 120 seconds of sustained load
while time.time() < end:
y = x @ x
torch.cuda.synchronize()
time.sleep(20) # cooldown
poller.terminate()
Why was the clock so low?
The first thing I noticed under load was the clock speed. It started around 810 MHz, then settled near 735 MHz and stayed there, dropping as low as 675 MHz — all while utilization sat at a steady 100%. That was the first surprise: the T4's rated boost clock is 1590 MHz, so the card was running at less than half its rated speed while reporting that it was fully busy.
At first I wondered whether the card could reach its boost clock at all. It could: the moment the workload stopped, two samples briefly showed 1590 MHz at 0% utilization. So the hardware was capable of the full clock — it just couldn't sustain it while doing heavy work. That told me something was actively holding the clock down.
The next place to look was power. The T4's power limit is 70 watts, and under load the card drew 69–70 watts the entire time — pinned against its cap. To stay inside that budget, the GPU was lowering its own clock. That explained the low clock: not idleness, but a power ceiling.
But there was a complication. Temperature climbed to 82 °C, and the lowest clock reading occurred at roughly the same moment — which looks exactly like thermal throttling. So I checked what the card itself reported. Under load, the driver returned sw_power_cap: Active, while every thermal-slowdown flag stayed Not Active, even at 82 °C. The card was not throttling because it had hit a temperature limit; it was throttling because it had hit the power limit.
Temperature still played a role, just an indirect one. The likely mechanism: as the chip heats up, electrical leakage current rises, so more of the fixed 70-watt budget is wasted as heat rather than spent on computation — which leaves less power for useful work, which forces the clock down to stay within budget. Heat feeds into the power limit rather than tripping a thermal limit of its own.
Why was the memory still occupied?
After the workload stopped, utilization dropped back to 0% — but the memory was not released. That led to two separate observations.
The first is that a GPU reporting 0% utilization is not necessarily free. Before the test, at cold idle, the card sat at 0% utilization, a low clock, about 10 watts, and no memory allocated. After the test, it also read 0% utilization — but the CUDA context was still alive, the memory was still allocated, the clock stayed higher, and power draw was almost three times that of cold idle. The two zeros look identical on a utilization graph and mean completely different things. On a shared GPU server, this is the job that appears idle while still holding memory and preventing anyone else from using the card.
The second observation was about how much memory was held. If we do the math, an 8000×8000 float32 tensor is roughly 64,000,000 elements × 4 bytes ≈ 244 MiB. There were two such tensors, so I expected about 488 MiB. But nvidia-smi reported 872 MiB — a difference of 385 MiB, which came from the CUDA context and the driver and library allocations loaded onto the card. The practical takeaway: a 4 GB model does not fit comfortably on a 4 GB GPU, because the framework and CUDA claim their share first.
Would I get the same answer twice?
To check whether my results were representative, I ran the whole experiment a second time on another Tesla T4 in the same environment. The results were not the same.
The second run settled at roughly 30% less sustained clock speed than the first, even though the GPU model, the workload, and the final temperature were all the same. Same card on paper, same test, meaningfully different numbers.
The likely reasons are all things that vary invisibly between cloud instances: a different physical GPU chip, a different host machine, different chassis airflow, and normal silicon variation from one chip to the next.
The lesson stayed with me more than any single number did: one benchmark run on a cloud GPU should not be presented as a universal result for that GPU model. The card you get is one instance among many, and characterizing the hardware properly means repeating the test across several instances before quoting a figure.
A note on how the monitoring was done
Getting the measurement itself right took one attempt to get wrong first. In Colab, cells run in sequence, not in parallel — so I couldn't put the workload in one cell and the monitor in another and expect them to run at the same time. My first attempt did exactly that, and the monitor only captured the GPU after the workload had already finished.
The fix was to run the monitoring tool as a separate operating-system process, launched from the same cell as the workload (the code above). That let nvidia-smi keep sampling in the background while the matrix multiplication ran in the foreground. Beyond the Colab quirk, this is also the more correct way to monitor in general: the measurement should not compete with the workload it is measuring — the same reason real monitoring agents run as separate processes alongside a service rather than inside it.
The takeaway
The most important lesson is that GPU health and performance are not the same thing as utilization. Judged by utilization alone, the card in this project looked completely healthy — a steady 100%. Only once power and clock were included did the real behavior appear: continuously busy, but running at a substantially reduced clock because it was constrained by its power budget. Utilization told me the GPU was working; it did not tell me how well.
And the second run added a caution on top of that: even a complete, correct measurement describes one instance at one moment. The same card model, same workload, gave a meaningfully different result on a different cloud instance. A single reading tells you what one GPU did — not what that GPU model does.