File Descriptor Transfer over Unix Domain Sockets

Data Transfer over UDS

Now that we’ve established that a Unix domain socket allows communication between two processes on the same host, it’s time to explore what kind of data can be transferred over a Unix domain socket.

Since a Unix domain socket is similar to network sockets in many respects, any data that one might usually send over a network socket can be sent over a Unix domain socket.

Furthermore, the special system calls sendmsg and recvmsg allow sending a special message across the Unix domain socket. This message is handled specially by the kernel, which allows passing open file descriptions from the sender to the receiver.

File Descriptors vs File Description

Note that I mentioned file descripTION and not file descripTOR. The difference between the two is subtle and isn’t often well understood.

A file descriptor really is just a per process pointer to an underlying kernel data structure called the file description. The kernel maintains a table of all open file descriptions called the open file table. If two processes (A and B) try to open the same file, the two processes might have their own separate file descriptors, which point to the same file description in the open file table.

So “sending a file descriptor” from one Unix domain socket to another with sendmsg() really just means sending a reference to the file description. If process A were to send file descriptor 0 (fd0) to process B, the file descriptor might very well be referenced by the number 3 (fd3) in process B. They will, however, refer to the same file description.

The sending process calls sendmsg to send the descriptor across the Unix domain socket. The receiving process calls recvmsg to receive the descriptor on the Unix domain socket.

Even if the sending process closes its file descriptor referencing the file description being passed via sendmsg before the receiving process calls recvmsg, the file description remains open for the receiving process. Sending a descriptor increments the description’s reference count by one. The kernel only removes file descriptions from its open file table if the reference count drops to 0.

sendmsg and recvmsg

The signature for the sendmsg function call on Linux is the following:

ssize_t sendmsg(
    int socket,
    const struct msghdr *message,
    int flags
);

The counterpart of sendmsg is recvmsg:

ssize_t recvmsg(
     int sockfd,
     const struct msghdr *msg,
     int flags
);

The special “message” that one can transfer with sendmsg over a Unix domain socket is specified by the msghdr. The process which wishes to send the file description over to another process creates a msghdr structure containing the description to be passed.

struct msghdr {
    void            *msg_name;      /* optional address */
    socklen_t       msg_namelen;    /* size of address */
    struct          iovec *msg_iov; /* scatter/gather array */
    int             msg_iovlen;     /* # elements in msg_iov */
    void            *msg_control;   /* ancillary data, see below */
    socklen_t       msg_controllen; /* ancillary data buffer len */
    int             msg_flags;      /* flags on received message */
};

The msg_control member of the msghdr structure, which has length msg_controllen, points to a buffer of messages of the form:

struct cmsghdr {
    socklen_t cmsg_len;    /* data byte count, including header */
    int       cmsg_level;  /* originating protocol */
    int       cmsg_type;   /* protocol-specific type */
    /* followed by */
    unsigned char cmsg_data[];
};

In POSIX, a buffer of struct cmsghdr structures with appended data is called ancillary data. On Linux, the maximum buffer size allowed per socket can be set by modifying /proc/sys/net/core/optmem_max.

Ancillary Data Transfer

While there are a plethora of gotchas with such data transfer, when used correctly, it can be a pretty powerful mechanism to achieve a number of goals.

On Linux, there are three such types of “ancillary data” that can be shared between two Unix domain sockets:

SCM_RIGHTS
SCM_CREDENTIALS
SCM_SECURITY

All three forms of ancillary data should only be accessed using the macros described below and never directly.

struct cmsghdr *CMSG_FIRSTHDR(struct msghdr *msgh);
struct cmsghdr *CMSG_NXTHDR(struct msghdr *msgh, struct cmsghdr *cmsg);
size_t CMSG_ALIGN(size_t length);
size_t CMSG_SPACE(size_t length);
size_t CMSG_LEN(size_t length);
unsigned char *CMSG_DATA(struct cmsghdr *cmsg);

While I’ve never had a need to use the latter two, SCM_RIGHTS is what I hope to explore more in this post.

SCM_RIGHTS

SCM_RIGHTS allows a process to send or receive a set of open file descriptors from another process using sendmsg.

The cmsg_data component of the cmsghdr structure can contain an array of the file descriptors that a process wants to send to another.

struct cmsghdr {
    socklen_t cmsg_len;    /* data byte count, including header */
    int       cmsg_level;  /* originating protocol */
    int       cmsg_type;   /* protocol-specific type */
    /* followed by */
    unsigned char cmsg_data[];
};

The receiving process uses recvmsg to receive the data.

The book The Linux Programming Interface has a good programmatic guide on how to use the sendmsg and recvmsg.

参考资料:

文章目录

Data Transfer over UDS

File Descriptors vs File Description

sendmsg and recvmsg

Ancillary Data Transfer

SCM_RIGHTS