AMD GPU Servers: IOMMU / ACS Effects on PCIe Peer-to-Peer Traffic

Problem and scope

The source describes AMD GPU servers with PCIe switch chips where enabled IOMMU/ACS settings redirect peer-to-peer traffic through the CPU root complex, reducing bandwidth or causing a hang. ACS means Access Control Services. IOMMU and ACS are related configuration factors, not interchangeable names for one feature.

This is a model-specific Linux/CentOS case. Current NVIDIA guidance distinguishes bare-metal PCIe P2P, GPUDirect use cases and virtualization. Do not treat this historical recipe as a universal setting: translated IOMMU mode can be unsuitable for bare-metal GPU P2P, while virtual machines require ACS/isolation. Validate the actual platform/workload before changing these controls.

1. Boot parameters in the source

The vendor adds amd_iommu=on iommu=pt to GRUB_CMDLINE_LINUX in /etc/default/grub. Append to and preserve the existing required kernel arguments; do not replace them with only the two flags. The following is the source’s illustrative configuration:

GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="amd_iommu=on iommu=pt"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true

Here iommu=pt requests a passthrough default domain. Confirm the active GPU-path configuration against the matching vendor guidance; seeing an IOMMU-related log line is not proof that PCIe P2P is correctly configured.

The source’s GRUB-generation paths are:

# Legacy boot, source CentOS layout
grub2-mkconfig -o /boot/grub2/grub.cfg

# UEFI boot, source CentOS layout
grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg

These paths depend on distribution/version and boot mode. Use the actual system’s documented output path. The case then reboots and checks:

reboot
dmesg | grep -i IOMMU

The full example output from the stored source is retained:

[    1.589058] pci 0000:66:00.2: AMD-Vi: IOMMU performance counters supported
[    1.589085] pci 0000:3b:00.2: AMD-Vi: IOMMU performance counters supported
[    1.589102] pci 0000:2a:00.2: AMD-Vi: IOMMU performance counters supported
[    1.589115] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    1.589144] pci 0000:f0:00.2: AMD-Vi: IOMMU performance counters supported
[    1.589169] pci 0000:c6:00.2: AMD-Vi: IOMMU performance counters supported
[    1.589192] pci 0000:ad:00.2: AMD-Vi: IOMMU performance counters supported
[    1.589211] pci 0000:7a:00.2: AMD-Vi: IOMMU performance counters supported
[    1.880060] pci 0000:66:00.2: AMD-Vi: Found IOMMU cap 0x40
[    1.880079] pci 0000:3b:00.2: AMD-Vi: Found IOMMU cap 0x40
[    1.880086] pci 0000:2a:00.2: AMD-Vi: Found IOMMU cap 0x40
[    1.880091] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    1.880096] pci 0000:f0:00.2: AMD-Vi: Found IOMMU cap 0x40
[    1.880101] pci 0000:c6:00.2: AMD-Vi: Found IOMMU cap 0x40
[    1.880106] pci 0000:ad:00.2: AMD-Vi: Found IOMMU cap 0x40
[    1.880111] pci 0000:7a:00.2: AMD-Vi: Found IOMMU cap 0x40
[    1.888800] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[    1.888866] perf/amd_iommu: Detected AMD IOMMU #1 (2 banks, 4 counters/bank).
[    1.888905] perf/amd_iommu: Detected AMD IOMMU #2 (2 banks, 4 counters/bank).
[    1.888948] perf/amd_iommu: Detected AMD IOMMU #3 (2 banks, 4 counters/bank).
[    1.888993] perf/amd_iommu: Detected AMD IOMMU #4 (2 banks, 4 counters/bank).
[    1.889046] perf/amd_iommu: Detected AMD IOMMU #5 (2 banks, 4 counters/bank).
[    1.889091] perf/amd_iommu: Detected AMD IOMMU #6 (2 banks, 4 counters/bank).
[    1.889129] perf/amd_iommu: Detected AMD IOMMU #7 (2 banks, 4 counters/bank).

To assess the current mode, also inspect /proc/cmdline and the platform’s domain/mapping messages. The source’s “any output means success” test only detects matching log text.

2. ACS script in the source

The article provides ACS_Disable.zip. Its ACS_Disable.sh was inspected for this translation: it enumerates PCI devices and writes zero to ACS control on every device where that capability can be read. It is not limited to one identified GPU switch.

Only for the validated bare-metal configuration and approved vendor procedure, the source runs:

sh ACS_Disable.sh

It then adds this startup line, with the script’s actual absolute path:

bash /file_path/ACS_Disable.sh

The original uses /etc/rc.d/rc.local, or /etc/rc.local where appropriate, and:

chmod +x /etc/rc.d/rc.local
# Or, for the alternative path used by the system:
chmod +x /etc/rc.local

Confirm that the intended startup mechanism is enabled, then verify after reboot. Do not apply the ACS-disabling script to a virtualized/isolation-dependent system by analogy. Follow the current NVIDIA GPU P2P / ACS guidance and the server vendor’s matching instructions.