本文将mark下virtio-net的Interrupt Coalescing技术相关notes。

SPEC

feature bit

spec中VIRTIO_NET_F_VQ_NOTF_COAL和VIRTIO_NET_F_NOTF_COAL这两个feature bit是Interrupt Coalescing相关特性。

VIRTIO_NET_F_VQ_NOTF_COAL(52) Device supports virtqueue notification coalescing.
VIRTIO_NET_F_NOTF_COAL(53) Device supports notifications coalescing.

Coalescing parameters

virtio-net通过control queue来配置Interrupt Coalescing的参数，详情可参见”Notifications Coalescing”一节。

1
2
3
4
struct virtio_net_ctrl_coal {
    le32 max_packets;
    le32 max_usecs;
};

Coalescing parameters:

  • max_usecs for RX: Maximum number of microseconds to delay a RX notification.
  • max_usecs for TX: Maximum number of microseconds to delay a TX notification.
  • max_packets for RX: Maximum number of packets to receive before a RX notification.
  • max_packets for TX: Maximum number of packets to send before a TX notification.

Operation

The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed.

When the device has non-zero max_usecs and non-zero max_packets, it starts counting microseconds and packets upon receiving/sending a packet. The device counts packets and microseconds for each receive virtqueue and transmit virtqueue separately. In this case, the notification conditions are met when max_usecs microseconds elapse, or upon sending/receiving max_packets packets, whichever happens first. Afterwards, the device waits for the next packet and starts counting packets and microseconds again.

When the device has max_usecs = 0 or max_packets = 0, the notification conditions are met after every packet received/sent.

virtqueue coalescing moderation

virtio-net: support the virtqueue coalescing moderation

virtio_net: add per queue interrupt coalescing support

Currently, coalescing parameters are grouped for all transmit and receive virtqueues. virtqueue coalescing moderation supports setting or getting the parameters for a specified virtqueue, and a typical application of this function is netdim.

When the traffic between virtqueues is unbalanced, for example, one virtqueue is busy and another virtqueue is idle, then it will be very useful to control coalescing parameters at the virtqueue granularity.

DIM

virtio-net already supports per-queue moderation parameter setting. Based on this, we use the netdim library of linux to support dynamic coalescing moderation for virtio-net.

Net DIM算法通过统计当前网络中单个队列的流量信息和中断次数，自适应计算中断调整方向和步长，并将结果配置下发到设备，以达到提升网络吞吐量的目的。virtqueue coalescing moderation 可以让 virtio-net支持动态中断调节，并对逐个队列下发中断调节参数。

Dynamic Interrupt Moderation (DIM) (in networking) refers to changing the interrupt moderation configuration of a channel in order to optimize packet processing. The mechanism includes an algorithm which decides if and how to change moderation parameters for a channel, usually by performing an analysis on runtime data sampled from the system. Net DIM is such a mechanism. In each iteration of the algorithm, it analyses a given sample of the data, compares it to the previous sample and if required, it can decide to change some of the interrupt moderation configuration fields. The data sample is composed of data bandwidth, the number of packets and the number of events. The time between samples is also measured. Net DIM compares the current and the previous data and returns an adjusted interrupt moderation configuration object. In some cases, the algorithm might decide not to change anything. The configuration fields are the minimum duration (microseconds) allowed between events and the maximum number of wanted packets per event(笔者注：在virtio-net中，configuration fields就是max_usecs和max_packets，感觉是maximum duration allowed between events才对). The Net DIM algorithm ascribes importance to increase bandwidth over reducing interrupt rate.

参考资料:

  1. virtio v1.3
  2. 高性能网络SIG月度动态：virtio-net 支持动态中断调节，SMC v2 协议增加新扩展
  3. 高性能网络SIG月度动态：virtio 支持动态中断聚合，SMCv2.1协议正式发布
  4. 高性能网络SIG月度动态：virtio 动态中断调节优化、多项内核网络缺陷修复
  5. virtio-net: support the virtqueue coalescing moderation
  6. Net DIM - Generic Network Dynamic Interrupt Moderation
  7. DYNAMICALLY-TUNED INTERRUPT MODERATION (DIM)
  8. Broadcom Ethernet Network Adapter User Guide:Interrupt Moderation