A passive GPU in a tower: VRAM hit 106 °C while the core throttled

The first time I put a real load on my new 32 GB GPU, the hotspot went from 59 °C to 100 °C in about 70 seconds, and it was still climbing when my test script aborted. There was no plateau. There was also no fan to turn up: the card is a passive, server-style board, and Linux exposed no fan sensor at all.

It took a day of tests to make it safe for unattended AI work, and the final surprise came a week later: the part that overheated was not the GPU core. It was the memory, and some of its heat came from the CPU.

This post is the full set of measurements, the levers that worked, the ones that only looked like they worked, and what I would check on any GPU before trusting it with overnight jobs.

Contents · 12 sections

TL;DR #

• My card (an AMD Radeon AI PRO R9700, short ASRock board) is built for server airflow. In a tower case with no forced air through its fins, it heated linearly with no plateau.
• When the core hits its limit, the GPU throttles the core, but not the memory. VRAM kept rising, to 106 °C in one test, close to the 108 °C slowdown point.
• Some settings are accepted but not enforced. The overdrive memory-clock limit read back correctly and changed nothing under load. Measure clocks, do not trust what sysfs echoes back.
• What worked: an external fan forced through the fins, the 210 W power cap (the lowest the firmware accepts), and later a manual memory-clock limit. Together they cost about 3 to 7% of generation speed.
• The memory chips sit on the back of the board with no backplate cooling, where the CPU’s heat rises onto them. CPU-heavy models raised GPU memory temperature even when the GPU was lightly used.

The card and the case #

The workstation is a dual-socket Xeon tower. The new card replaced an 8 GB consumer GPU that had its own fans. The R9700 board I bought is a short, blower-less design that expects a server chassis to push air through it.

The first warning sign was in sysfs: amdgpu exposed no fan sensor, no PWM control and no fan curve. A later look with overdrive enabled showed a fan-control directory with an all-zero curve and still no fan reading. Consistent with a passive card.

The load tests #

Each test used the same workload: a 14B language model generating continuously, with a script that logged edge, hotspot (junction) and memory temperature, power and clocks every second, and stopped at a safety limit.

Test	Change	Time to 100 °C hotspot	Notes
1	as installed	~70 s	VRAM 92 °C, 29 tok/s
2	cooler/airflow change	~2 min 20 s	43-44 tok/s, idle cooldown ~2x faster
3	second change	~1 min 55 s	slightly worse than 2
4	paper duct	~2 min 10 s	lower power, so not comparable raw
5	another layout	~2 min 30 s	better than 4, not better than 2

Comparing tests by time-to-limit was misleading, because average power differed (138 W in one test, 206 W in another). Normalised, the heating rate per 100 W told the real story: 0.156, 0.171, 0.197 and 0.166 °C per second for tests 2 to 5. Every change after test 2 made cooling worse.

Lesson one: normalise by average power when you compare thermal runs, or you will reward a change for making the card do less work.

What throttling does, and what it does not #

Test 7 let the card reach its own limit, with a stop at hotspot 104 °C or VRAM 106 °C:

• The core self-throttled: hotspot held at 100 to 101 °C, the core clock fell from about 3.1 GHz to 1.7 to 2.5 GHz, power dropped from about 290 W to 210 to 235 W, and generation fell from 44 to 38 tokens per second.
• The memory did not: VRAM kept rising from 94 to 106 °C in about 45 seconds, and the test stopped about 2.5 minutes into the load.

Die throttling does not reduce the memory clock. The next protection for the memory would have been the slowdown at 108 °C, then shutdown at 113 °C. That made unattended load unsafe at the airflow I had.

Settings that are accepted but not enforced #

I tried to cap the card in software. This is where sysfs lied politely.

• The power cap (power1_cap) has a floor of 210 W on this card (range 210 to 300 W). There is no low-power option.
• Forcing the core to its lowest DPM level (power_dpm_force_performance_level=manual, pp_dpm_sclk set to the 500 MHz level) did not hold under load: the core read 2,061 MHz mid-test.
• With overdrive enabled through amdgpu.ppfeaturemask, the file pp_od_clk_voltage appeared with a core offset, a memory-clock maximum and a voltage offset. I set the memory maximum to 1,000 MHz.

To check, I measured with amd-smi once per second in an A/B run of 75 seconds each:

	Defaults	Overdrive limits
Average power	207 W	182 W
Average core clock	2,684 MHz	2,396 MHz
Memory clock	1,258 MHz in 49/51 samples	1,258 MHz in 45/47 samples
Tokens/s	43.9	42.8 to 44.9

The core offset and power cap were enforced (12% less power, 11% lower core clock). The memory limit was accepted, reported back, and ignored. Memory temperature was the limiting factor, and overdrive could not touch it.

A side lesson about evidence: pp_dpm_mclk shows the DPM level labels, not a measured clock, so my first reading of it was weak evidence. The better evidence was that tokens per second did not change with a “20% memory cut”. Use a tool that reports measured clocks.

What worked #

Air through the fins #

A CPU fan on an external 24 V supply, pushing air straight through the card’s fins, changed everything. Idle temperature dropped from 40 to 43 °C to 29 °C. In an 8.5-minute full-load test at default clocks (average 265 W, 49 to 50 tokens/s, faster than before because nothing throttled), the card reached its first plateau: edge 80 °C, hotspot 96 °C, VRAM 102.5 °C from minute 3. But only about 5 °C of margin to the memory slowdown point.

Plus the 210 W cap #

Same fan, power capped at 210 W, 8 minutes of full load: average 204 W, 47.9 to 48.6 tokens/s, about 3% slower. Plateau from minute 5 at edge about 68 °C, hotspot about 79 °C and VRAM about 88 °C (maximum 90 °C). That is 19 °C of memory margin for 3% of speed, the best trade of the day.

The cap resets on every boot, so a oneshot service sets it before the inference services start:

#!/bin/sh
CAP_W=210
for n in /sys/class/hwmon/hwmon*/name; do
  [ "$(cat "$n")" = amdgpu ] || continue
  h=$(dirname "$n")
  min=$(( $(cat "$h/power1_cap_min") / 1000000 ))
  [ "$CAP_W" -lt "$min" ] && CAP_W=$min
  echo $(( CAP_W * 1000000 )) > "$h/power1_cap"
done

The inference services Require this unit, so they never start on an uncapped card.

A manual memory-clock limit, for the dense model #

A week later I moved my heavy model from a 35B mixture-of-experts model (experts on the CPU) to a dense 27B model fully on the GPU. Even at 210 W, GPU memory reached 97 to 102 °C under the 27B.

This time the memory-clock limit did work, through a different path: manual DPM mode with the allowed memory-clock levels restricted (pp_dpm_mclk limited to the levels up to 875 MHz), applied at boot. Measured with a 3-minute 27B load beside the always-on 9B model:

	Memory °C	Hotspot	Power	Prompt tok/s	Generation tok/s
Stock at 210 W	97 / 99	94 °C	207 W	410	15.3
Memory clock ≤ 875 MHz	85 / 88	86 °C	173 W	408	14.3

About 12 °C cooler memory and 35 W less, for about 7% slower generation. Prompt processing barely changed, which makes sense: prefill is compute-bound, generation is memory-bound.

The CPU heats the GPU’s memory #

The last finding was the least obvious. On this board the memory chips sit on the back of the card, with no backplate or heatsink on that side, and heat from the CPUs rises onto them. When I ran the 35B model with its experts on the CPU (44 threads busy), GPU memory reached 90 to 92 °C even though the GPU itself was lightly used.

I had argued that “the 35B only loads the CPU, so it is harmless for the GPU”. That was wrong. CPU load is GPU memory heat on this machine. Two practical consequences:

• Prefer GPU-resident models over CPU-offloaded ones, for thermals as well as speed.
• The acute problem went away after I lowered the system RAM frequency; since then the temperatures have been fine.

Fans on the host matter too #

While testing the GPU, one CPU socket sat at 93 °C, its critical point, with about 360,000 core throttle events in 71 minutes. The cause was my own fan configuration. On this Dell workstation the fan driver offers only three levels (off, low about 1,000 rpm, high about 5,700 rpm), and it rounds a requested duty to the nearest level: 75% became LOW, 76% became HIGH. I had set 75%.

Re-enabling BIOS fan control and removing the fixed-speed rules dropped that socket from 93 °C to 55 °C, with zero throttle events in 90 seconds while the GPU drew 244 W. If your fan control software offers a percentage, check what the hardware actually does with it.

Monitoring and guards #

node_exporter already exposes the amdgpu hwmon sensors (node_hwmon_temp_celsius with edge, junction and mem labels, plus critical values of 100, 100 and 108 °C on this card). My alert rules: temperature within 8 °C of critical, power cap not applied (above 215 W), and sustained high power.

One more bug worth sharing: a batch OCR worker had a heat guard that read the hwmon directory in a single try block. An I/O error on a non-temperature file (a memory-clock frequency input) aborted the whole loop, so the guard saw 0 °C and never paused. Read only the temp* sensors, each in its own try.

Checklist #

• Before overnight jobs, run a load test and confirm the temperature plateaus. No plateau means no safe steady state.
• Log edge, hotspot and memory temperature. The memory may be your limit, and core throttling will not protect it.
• Normalise thermal comparisons by average power.
• Verify every limit with measured clocks and power, not the value sysfs reads back.
• Set the power cap at boot and make inference services depend on it.
• On passive cards, force air through the fins.
• Count CPU load as a heat source for the GPU if the board has exposed memory on its back.
• Check that fan percentages map to real fan speeds on your hardware.

Frequently asked questions #

Why does my GPU memory overheat when the core is throttling? #

Core throttling lowers the core clock and power, but not the memory clock. Under a memory-bound load such as LLM generation, the memory keeps working at full speed and keeps heating. You need a memory-clock limit, a lower power cap or more airflow.

Can I lower the power limit of an AMD GPU below its minimum on Linux? #

Not through power1_cap; the firmware rejects values below power1_cap_min (210 W on my card). Clock limits through manual DPM levels are the next lever, but verify that they are enforced under load.

How much speed does a power or memory-clock cap cost for LLMs? #

On my card, the 210 W cap cost about 3% of generation speed, and limiting memory clock to 875 MHz cost about 7% more, while prompt processing barely changed. In return, memory temperature dropped by 14 °C and 12 °C respectively.

Can CPU load really raise GPU temperature? #

Yes, if heat from the CPU area reaches the GPU board. On my workstation the GPU’s memory chips have no cooling on the back of the board, and CPU-heavy inference raised GPU memory to 90 to 92 °C while the GPU core was lightly used.

The lesson #

The number that mattered was never the one I looked at first. Not the core temperature, but the memory. Not the setting I wrote, but the clock I measured. Not the GPU’s own load, but the CPU’s. Instrument everything, test until you see a plateau, and treat any limit as unproven until a measurement under load shows it working.