Swapping an 8 GB NVIDIA card for a 32 GB AMD GPU: what broke

In late September I replaced the only GPU in my home lab, an 8 GB NVIDIA RTX 4060 Ti, with a 32 GB AMD Radeon AI PRO R9700. Four times the VRAM for local AI. The card itself went in without drama. Everything around it took about two days to settle: a network interface that changed its name twice, an inference server that silently fell back to the CPU, a 4 GiB allocation limit nobody warned me about, and Kubernetes pods still asking for a GPU that no longer existed.

This post is what broke, in the order it broke, and what got better, with the numbers.

Contents · 7 sections

TL;DR #

• Plan for side effects outside the GPU stack: moving cards renumbered the PCI bus and renamed the onboard network interface, which broke a video-conference bridge and a LAN route.
• When the GPU disappears, inference does not stop. It falls back to the CPU, and on my workstation that meant load 117 and tens of thousands of CPU thermal-throttle events.
• On AMD, Ollama’s Vulkan backend caps one allocation at 4 GiB, so a model with a larger buffer quietly runs on the CPU. The ROCm backend has no such cap.
• CUDA_VISIBLE_DEVICES=-1 also hides an AMD GPU from HIP. Remove NVIDIA-era environment variables.
• What got better: models that never fit before, and throughput. The pipeline model now runs at 67 tokens/s, and a batch workload went from 468 to 2,004 items per hour across the backends I tested.

The starting point #

The machine is a dual-socket Xeon workstation with 88 threads, PCIe 3.0, and a 1300 W power supply. The 4060 Ti was shared by OCR, speech-to-text, a fine-tuning UI, Jupyter and image generation, one heavy user at a time. Ollama served language models on the CPU, because 8 GB was not enough for the models I wanted.

Before the swap I did the boring preparation, and it paid off:

• confirmed the card (gfx1201, RDNA4) was on the ROCm compatibility list;
• installed the ROCm user-space libraries without the DKMS kernel module (amdgpu-install --usecase=rocm --no-dkms), because the in-kernel amdgpu driver was new enough;
• backed up every configuration that mentioned the GPU;
• wrote a runbook with checks and a rollback for each step.

What broke #

1. The NVIDIA module vanished first, before the new card was even in #

While I waited for a power adapter I reseated the old card, and the desktop came up at 1024x768 on a simple framebuffer. Not the swap’s fault: unattended upgrades had installed a new kernel, and the prebuilt NVIDIA module package for it had not followed. No nvidia.ko existed for the running kernel. Installing the matching module package fixed it.

Lesson: after every kernel update, check that out-of-tree GPU modules moved to the new kernel ABI.

2. The network interface was renamed, twice #

The onboard Intel NIC had been enp9s0 for 171 days. After I removed and reinserted GPUs it became enp6s0, and after the new card went in, enp11s0. Predictable interface names follow the PCI bus number, and adding a card renumbered the bus.

Two things keyed on the old name broke silently: a WebRTC video bridge that binds only to an allowed interface (it crash-looped with “Failed to bind single-port”), and a NetworkManager dispatcher script that adds a LAN route.

The fix was to accept all known names, and the lasting rule is: after any PCI change, grep -r your configuration for interface names, or pin the name with a MAC-based .link file.

3. Inference silently moved to the CPU #

With the NVIDIA card gone, Ollama found no GPU it could use and ran everything on the CPU. It did not fail. It used about 41 cores, the load average reached 117, embedders timed out and their pods crash-looped, and a re-embedding job stalled.

The CPU coolers were not built for that. One package logged 84,343 thermal-throttle events in 37 minutes. My rule since then: never let LLM work fall back to the CPU silently. Alert on it.

4. Vulkan’s 4 GiB allocation limit #

The Ollama build I had first used the Vulkan backend on AMD. It worked for most models, but one model with a 4.4 GiB tensor buffer landed 5.4 GiB on the CPU instead. The reason is a Vulkan device limit: maxMemoryAllocationSize was 0xfffffffc, just under 4 GiB, so no single buffer can be larger.

The ROCm build of the same Ollama version has no such limit. Before you choose a backend, check the largest buffer each of your models needs.

5. An NVIDIA environment variable hid the AMD GPU #

My first switch to the ROCm backend left Ollama CPU-only for about two minutes. An old drop-in still set CUDA_VISIBLE_DEVICES=-1 (to stop it from using the NVIDIA card). HIP honours that variable too, so the AMD card was hidden as well.

Related trap: in a ROCm container, passing -e HSA_OVERRIDE_GFX_VERSION= with an empty value made PyTorch report “No HIP GPUs are available”. On this card, omit the variable entirely.

6. systemd drop-ins apply in lexical order #

My tuning drop-in was named 40-tuning.conf. An older override.conf set the same variables and sorts later, so it won. Name tuning drop-ins zz-*.conf and verify the effective values in the server’s startup log, not in the file you edited.

7. A “faster” setting broke the embedding model #

I forced flash attention and an 8-bit KV cache for all Ollama models. Chat models were fine. The bge-m3 embedding model, with an 8192-token batch, could no longer allocate its compute buffer: 238 failed embedding requests in about 20 minutes, and library search degraded. Ollama’s automatic mode had been choosing per model, and that was better. Test every model class after a server-wide change.

8. Monitoring went blind #

My ai monitoring script was built on nvidia-smi. On AMD, per-process VRAM is harder: amd-smi process lists only ROCm/KFD compute processes, so Vulkan clients showed nothing. The working source is /proc/<pid>/fdinfo, which has drm-memory-vram per DRM client. One catch: reading /proc/*/cmdline for every process can hang on processes stuck in uninterruptible sleep (NFS), so only read it for processes that have a /dev/dri file open.

Card-level numbers come from sysfs (mem_info_vram_used) and hwmon (edge, junction and memory temperatures, power).

9. Leftovers kept running #

• An NVIDIA container-toolkit refresh unit looped, running nvidia-smi 1,410 times and logging “No NVIDIA GPU found”. Mask it.
• Kubernetes pods with nvidia.com/gpu limits or the nvidia runtime class sat Pending or crash-looped. I moved speech-to-text, video transcoding and the RAG app to CPU images and deleted the NVIDIA device-plugin and DCGM exporter daemonsets.
• CUDA-only workloads (an OCR service, Jupyter, the fine-tuning UI) were scaled to zero until ROCm or CPU builds exist. Fine-tuning on this card is still on that list.
• Hard power-offs during testing corrupted the tail of a Redis append-only file; redis-check-aof --fix repaired it after a backup.

10. OCR needed tuning on ROCm #

A document-conversion service rebuilt with PyTorch for ROCm worked, but its OCR model asked for a 9.56 GiB GEMM workspace. Disabling GEMM convolutions avoided that but ran at 30 to 62 seconds per page. Allowing GEMM with a memory fraction cap, MIOpen’s fast find mode and expandable segments gave 8.9 seconds per page, with text identical to the old CUDA output.

# environment that worked for the OCR container on ROCm
MIOPEN_FIND_MODE=FAST
PYTORCH_ALLOC_CONF=expandable_segments:True
# plus a per-process memory fraction (about 0.45 of the card)

What got better #

Everything that motivated the swap, and more.

Model sizes. With 32 GB, a 9B pipeline model with a speculative-decoding head runs fully on the GPU at 67 tokens/s for one stream, and a 27B dense model fits beside it on demand. A 35B mixture-of-experts model fully on the GPU reached 60 tokens/s in a test.

Throughput, on one batch workload (writing study-guide sections from book chapters, 32 real sections, same prompt):

Backend and setup	Sections per hour
Ollama, gemma3 4B, 2 concurrent	468
llama.cpp, gemma3 4B Q4, 16 concurrent, default batch	1,009
llama.cpp, tuned batch (-b 4096 -ub 2048), 16 concurrent	1,364
vLLM, gemma3 4B bf16, 16 concurrent	2,004
llama.cpp, Qwen3.5 9B with MTP, 16 concurrent	1,468

vLLM was the fastest, but it held 14.7 GiB (llama.cpp at default batch: about 8 GB; tuned: about 12.9 GB) and could not coexist with the other models I needed, so I chose llama.cpp. The 9B row is the one that matters: a much better model than the 4B, faster than any 4B setup except vLLM, and 32 of 32 answers valid JSON.

The costs are real too. The card is a passive, blower-less server design in a tower case, and its thermals needed a full day of testing and an extra fan. That is a separate post.

Checklist #

• Check the card is on your ROCm version’s compatibility list; install user space without DKMS if your kernel driver is new enough.
• Back up every config that mentions the GPU and write a rollback.
• After the swap, grep for network interface names; pin them by MAC if you can.
• Alert when inference falls back to the CPU.
• Remove NVIDIA-era variables, especially CUDA_VISIBLE_DEVICES=-1; never pass empty HSA_OVERRIDE_GFX_VERSION.
• Check the largest buffer per model against backend limits (Vulkan: 4 GiB per allocation).
• Test every model class (chat, embeddings, vision) after changes.
• Rewrite monitoring for sysfs/hwmon and /proc/<pid>/fdinfo.
• Remove NVIDIA daemonsets, runtime classes and GPU limits from Kubernetes manifests.

Frequently asked questions #

Does Ollama work on AMD GPUs? #

Yes, with the ROCm build or the Vulkan backend. Watch the Vulkan limit: a single allocation cannot exceed about 4 GiB, so models with larger buffers fall back to the CPU. The ROCm build did not have that limit on my card.

Why did my network stop working after adding a GPU? #

Linux predictable interface names include the PCI bus number. Adding or moving a card can renumber the bus and rename the onboard NIC. Any config that names the interface breaks until you update it or pin the name by MAC address.

Is llama.cpp or vLLM better on an AMD RDNA4 card? #

In my test vLLM had about twice the throughput of llama.cpp at default settings, but needed 14.7 GiB against about 8 GB. Tuning llama.cpp’s batch closed part of the gap at about 12.9 GB. On a single shared GPU running several models, llama.cpp’s flexibility won.

How do I see per-process VRAM on AMD? #

Read drm-memory-vram in /proc/<pid>/fdinfo/<fd> for each DRM file descriptor. amd-smi process shows only ROCm compute processes, not Vulkan clients.

The lesson #

A GPU swap is not a GPU change. It is a PCI change, a driver change, a backend change and a monitoring change at once, and every one of them can fail quietly. The worst failures were the silent ones: a renamed interface, a CPU fallback, an allocation cap. Prepare a runbook, change one layer at a time, and after each step check not only that things run, but that they run where you think they run.