本文将会介绍Linux中pin住文件page cache的方法。

vmtouch

-t
Touch virtual memory pages. Reads a byte from each page of the file. If the page is not resident in memory, a page fault will be generated and the page will be read from disk into the file system’s memory cache.
-l
Lock pages into physical memory. This option works the same as “-t” except it calls mlock(2) on all the memory mappings and doesn’t close the descriptors when finished. The files will be locked in physical memory until the vmtouch process is killed.

page cache

For read reuqest:

  1. When a user-space application wants to read data from disks, it asks the kernel for data using special system calls such as read(), pread(), vread(), mmap(), sendfile(), etc.

  2. Linux kernel, in turn, checks whether the pages are present in Page Cache and immediately returns them to the caller if so. As you can see kernel has made 0 disk operations in this case.

  3. If there are no such pages in Page Cache, the kernel must load them from disks. In order to do that, it has to find a place in Page Cache for the requested pages. A memory reclaim process must be performed if there is no free memory (in the caller’s cgroup or system). Afterward, kernel schedules a read disk IO operation, stores the target pages in the memory, and finally returns the requested data from Page Cache to the target process. Starting from this moment, any future requests to read this part of the file (no matter from which process or cgroup) will be handled by Page Cache without any disk IO operation until these pages have not been evicted.

由此可知，mmap() file reads也是会使用到page cache的。

mlock

1
int mlock(const void addr[.len], size_t len);

mlock(), mlock2(), and mlockall() lock part or all of the calling process’s virtual address space into RAM, preventing that memory from being paged to the swap area.

实际上内核对mlock锁住的页面主要做了两步比较重要的操作:

  1. 调用mlock的时候就将所需要的物理页面准备好
  2. 内存回收时，当扫描到相关的物理页面时，将其放入不可回收的lru链表

第一步保证访问的虚拟地址对应的物理页面在内存中，第二步保证了锁住的页面不会被回收。

vmtouch source code

1
2
3
4
5
6
7
8
9
10
// https://github.com/hoytech/vmtouch/blob/master/vmtouch.c
void vmtouch_file(char *path) {
    ...
    fd = open(path, open_flags, 0);
    ...
    mem = mmap(NULL, len_of_range, PROT_READ, MAP_SHARED, fd, offset);
    ...
    mlock(mem, len_of_range)
    ...
}

实现其实也比较简单，基于mmap和mlock即可！

参考资料:

  1. vmtouch.c
  2. man vmtouch
  3. man mlock
  4. Essential Page Cache theory
  5. Page Cache and basic file operations
  6. mlock锁原理剖析