Notes about PCIe resize bar

本文将mark下PCIe resize bar的相关notes。

Overview

Resizable BAR support is a PCIe extension that allows resizing a PCIe device’s mappable memory/register space. An important use case are GPUs. GPUs generally declare only have 256MiB worth of BARs mapping GPU memory to maintain compatibility with 32bit operating systems. However, for performance (particularly when using PCIe P2P), it is desirable to be able to map the entirety of GPU memory, necessitating resizable BARs.

Case:GPU game

When a game is running, the CPU needs access to a graphic card’s memory, and by default it can only access 256MB at a time. However, resizable BAR removes the size limitation so the CPU has access to all of a GPU’s memory, theoretically allowing for improved performance. This adjustment lets the CPU request much larger chunks of data from the GPU, which can improve performance in games that can benefit from this change.

Resizable BAR lowers the overhead involved in the transfer of game assets between the CPU and GPU, sending fewer, larger transfers instead of numerous smaller transfers, and it can also allow multiple requests to happen at the same time instead of entering a queue. As an example, if your CPU needs 1GB of data it would have to make four requests using a 256MB limitation, or just one using Resizable BAR .

PCIe Spec details

The Resizable BAR Capability structure defines a PCI Express Extended Capability, which is located in PCI Express Extended Configuration Space, that is, above the first 256 bytes, and is shown below in Figure 7-127. This structure allows devices with this capability to be identified and controlled. A Capability and a Control register is implemented for each BAR that is resizable. Since a maximum of six BARs may be implemented by any Function, the Resizable BAR Capability structure can range from 12 bytes long (for a single BAR) to 52 bytes long (for all six BARs).