Scalable Work Submission in Device Virtualization

本文将以Intel Data Streaming Accelerator为例，讲解DMWr (Deferrable Memory Write) TLP、ENQCMD/ENQCMDS指令、ENQCMD Virtualization、SVA Work Submission In Guest相关内容。

1. DMWr TLP

1.1 What

Deferrable Memory Write (DMWr) transactions are a new type of TLP supported by the PCI Specifications. This new feature allows the completer to return an acknowledgement to the requester of the DMWr transaction and provides the completer a mechanism to temporarily refuse the request.

The Deferrable Memory Write (DMWr) is an Optional Non-Posted Request that enables a scalable high-performance mechanism to implement shared work queues and similar capabilities. With DMWr, devices can have a single shared work queue and accept work items from multiple non-cooperating software agents in a non-blocking way.

读完上述定义后，或许对DMWr的理解不够深刻，接下来我们将以DSA的SWQ(Shared Work Queue)为例，阐述下为什么要有DMWr。

1.2 Why

DMWr is a 64-byte non-posted write that waits for a response from the device before completing. The device returns Success if the descriptor is accepted into the work queue, or Retry if the descriptor is not accepted due to WQ capacity or QoS.

正常写mmio寄存器是posted tlp，也就是说completer不会给requester返回报文。
DMWr是non-posted write tlp，这也为retry带来了可能！

2. ENQCMD/ENQCMDS指令

On Intel CPUs, DMWr is generated using the ENQCMD or ENQCMDS instructions. The ENQCMD and ENQCMDS instructions return the status of the command submission in EFLAGS.ZF flag; 0 indicates Success, and 1 indicates Retry.

SDM vol2中有这两个指令的详细描述。

ENQCMD 中destination offset参数的含义： enqueue registers, which are special device registers accessed using memory-mapped I/O (MMIO). 说白了，offset就是MMIO enqueue registers的location！

2.1 Example in DSA

static inline  unsigned int
enqcmd(void *dst, const void *src)
{
    uint8_t retry;
    asm volatile(".byte 0xf2, 0x0f, 0x38, 0xf8, 0x02\t\n"
                 "setz %0\t\n"
                 : "=r"(retry) : "a" (dst), "d" (src));
    return (unsigned int)retry;
}

...
while (enqcmd(wq_portal, &desc) && enq_retry++ < ENQ_RETRY_MAX) ;
if (enq_retry == ENQ_RETRY_MAX) {
    printf("ENQCMD retry limit exceeded\n");
    rc = EXIT_FAILURE;
    goto done;
}
...