Open Source Summit Latin America 2022

A fact of life, one that almost every computer user has to face at some point, is that file systems fail. Whether it is due a faulty hardware, a sudden power loss, or a kernel bug, users don't enjoy losing their data for no reason.

When it happens on a user-facing machine, the problem is usually found and troubleshooted by inspecting the kernel log. The user, after some grumpiness, can then start a recovery process, like fsck, replace the disk, or restore from backup. But this kind of simple monitoring simply doesn't scale to a data-center. When dealing with a fleet of machines, kernel logs are lost, printk messages change or get truncated, and, more often than not, system administrators are left without any warning until it is too late.

In this talk, I will present a new kernel mechanism for daemons to monitor file system health through a series of push notification events from the kernel, which are triggered as soon as a volume corruption is found by the file system code. This feature, merged in Linux 5.16, once picked up by distros, won't prevent errors from happening, but will allow system administrators to reliably watch their volumes, and act immediately when necessary.