nofscache

nofscache is a loadable kernel moduel used to to eliminating page caching effects for user applications.

Requirements

Linux kernel version >= 4.12
Having CONFIG_ADVISE_SYSCALLS=y in /boot/config-$(shell uname -r)
This module affects only 64-bit userspace programs since we didn’t patch the compatibility version of system calls (compat_sys_xyzzy()).

Installation

git clone https://github.com/ljishen/nofscache.git
cd nofscache
make install

If you keep seeing processes under

[INFO] Checking transition state for module no_fscache (update every 2s)...

[INFO] 2 tasks are not in patched state:
USER       PID   TID CMD
root      1040  1131 /usr/bin/lxcfs /var/lib/lxcfs/
root      1040  1133 /usr/bin/lxcfs /var/lib/lxcfs/

...

You may need to manually kill these processes because they are stopping the module from finishing the transition state from unpatched to patched. See livepatch consistency model for more details.

To uninstall the kernel module, use

make uninstall

Again, you may need to kill processes to help the module finish the transition state from patched to unpatched.

Limitations

There are four basic Linux I/O models,

Four Linux IO Models

This module fully supports synchronous blocking I/O, and implicitly supports synchronous non-blocking I/O and asynchronous blocking I/O. For asynchronous non-blocking I/O, this module supports POSIX AIO as it is a user-space implementation that actually calls blocking I/O interfaces. The module may support libaio as it mainly focuses on direct I/O. We hope to support the latest I/O interface, io_uring, but this module is not ready yet. Here is a really good article explaining the differences between these I/O models.

Performance Results

Performance Results of Module no_fscache

For tests without module installed, we used the memory controller (memory.limit_in_bytes) in cgroup to limit the amount of memory (including page cache) can be used by the fio process. The limitations are generally set to a low number, and you can find the exact value we used for different tests in the section Result Details below.
Each number is the average of 5 data points in steady-state.
Each test runs for 60 seconds. Here are the fio job files of all tests.
We used block size of 4KiB for sequential tests and 256KiB for random tests.

Result Details

References

Design Considerations of Eliminating External Caching Effects for MBWU Construction
https://cross.ucsc.edu/news/blog/mbwuconstruction_122010.html
Implementing a Kernel Module to Eliminating External Caching Effects
https://cross.ucsc.edu/news/blog/mbwuconstruction_012020.html