本文将mark下RDMA Device Memory相关notes。

Introduction

Device Memory is a verbs API that allows using on-chip memory, located on the device, as a data buffer for send/receive and RDMA operations. The device memory can be mapped and accessed directly by user and kernel applications, and can be allocated in various sizes, registered as memory regions with local and remote access keys for performing the send/ receive and RDMA operations. Using the device memory to store packets for transmission can significantly reduce transmission latency compared to the host memory.

Motivation

staging buffer: 暂存缓冲区

Concepts

思考

可以类比于NVMe的CMB，RDMA Device Memory以mmio的形式expose给host，目的是让RDMA直接使用Device Memory，无需DMA到host的内存，减少了PCIe TLP的交互。

相关paper

Sherman(SIGMOD’22) is the first RDMA-based system that leverages on-chip memory of commodity RDMA NICs.

参考资料:

  1. Device Memory
  2. On-device memory usage for RDMA transactions
  3. On DPDK and RDMA Related Software
  4. ibv_alloc_dm(3)