Notes about CRIU(Checkpoint/Restore In Userspace)

本文将mark下CRIU(Checkpoint/Restore In Userspace)的相关notes。

What

CRIU(Checkpoint/Restore In Userspace) is a Linux software. It can freeze a running container (or an individual application) and checkpoint its state to disk. The data saved can be used to restore the application and run it exactly as it was during the time of the freeze. Using this functionality, application or container live migration, snapshots, remote debugging, and many other things are now possible.

Introduction

CRIU is a software framework for transparent checkpointing and restoring of Linux processes. It enables live migration, snapshots, or accelerated start-up of processes and containers. To extract the user-space application state, CRIU uses conventional debugging mechanisms, like ptrace. However, to extract the state of process-specific kernel objects, CRIU depends on special Linux kernel interfaces.

During recovery, CRIU runs inside the target process and recreates all OS objects on behalf of the target. This way, CRIU utilises the available OS mechanisms to run most of the recovery without the need for significant kernel modifications. Finally, CRIU removes any traces of itself from the process.

CRIU can also restore the state of TCP connections, a crucial feature for live migration of distributed applications. The Linux kernel introduced a new TCP connection state, TCP_REPAIR, for that purpose. In this state, a user-level process can modify the send and receive message queues, get and set the message sequence numbers and timestamps, or open and close a connection without notifying the other side.

进程状态

CRIU checkpoint与restore时需要考虑的进程状态:

All of the threads running within the process, where they are executing, their priority, and their signal handling state.
A complete memory map of the process: which mappings exist at which addresses and the protections that apply to each.
A list of the process’s open files, including the actual files that have been opened, whether each was opened for read, write, or append access, the current file position, and the file-descriptor number.
Every open network connection, who the peer is, which protocol is in use, and any in-transit data.
The configuration of the namespaces in which the process is running.
Active file notifications, terminal configurations, active timers, and no end of other details.

CRIU for RDMA and GPU

更多学术paper，可以参考Academic Research

CRIB

CRIB (Checkpoint/Restore In eBPF) is a proposed eBPF-based system for checkpointing and restoring process state, offering improved performance, flexibility, and extensibility over traditional Linux kernel methods like procfs. It consists of user-space programs, eBPF programs, and kernel functions to facilitate efficient dumping and restoring of process data directly within the kernel.

CRIB is an innovative approach to process checkpointing and restoring, designed to address the limitations of existing kernel mechanisms. It leverages the power of eBPF to provide a more performant, flexible, and extensible solution for managing process states.

参考资料:

文章目录

What

Introduction

进程状态

CRIU for RDMA and GPU

CRIB