Recently, I haven't been in the mood to write any articles. The promised articles about Raft have also been delayed due to some issues. But I still wanted to write something, so I decided to write a short article to record a problem I encountered today regarding a Linux kernel parameter, and also take some notes.
Getting Started#
I'm not a big fan of Mac, so I use Manjaro as my development environment at home (here's an advertisement, it's a great distribution, ready to use out of the box, only 50 cents for the ad). And my code tools are the JetBrains suite and VSCode.
Today, when I opened Goland, I noticed that the IDE gave me this warning: External file changes sync may be slow: The current inotify(7) watch limit is too low.
So, as you all know, I have a compulsion to address these warnings, so I decided to look it up.
Let's Talk About It#
We often have the need to monitor changes in a file or a directory, such as creating or deleting files. Our usual approach may be to use a brute force polling method.
However, this approach has poor performance. So, is there any way to handle this?
Yes! Linux provides the corresponding API to handle this, and that's what we're going to talk about today: inotify.
According to the official description, inotify is actually quite simple:
The inotify API provides a mechanism for monitoring file system events. Inotify can be used to monitor individual files, or to monitor directories. When a directory is monitored, inotify will return events for the directory itself, and for files inside the directory.
In other words, inotify is used to monitor file system events. It can be used on individual files or directories. Changes to the monitored file or the files within the monitored directory are within the scope of monitoring.
After monitoring the corresponding file, inotify will return the following events:
-
IN_ACCESS: File is readable.
-
IN_ATTRIB: Metadata has changed.
-
IN_CLOSE_WRITE: File opened for writing was closed.
-
IN_CLOSE_NOWRITE: File not opened for writing was closed.
-
IN_CREATE: File/directory was created in the monitored directory.
-
IN_DELETE: File/directory was deleted in the monitored directory.
-
IN_DELETE_SELF: File/directory being monitored was deleted.
-
IN_MODIFY: File was modified.
-
IN_MOVE_SELF: File/directory being monitored was moved.
-
IN_MOVED_FROM: File/directory was moved out of the monitored directory.
-
IN_MOVED_TO: File/directory was moved into the monitored directory.
-
IN_OPEN: File was opened.
There are a total of 12 event types, which cover our common needs. However, inotify also has its own drawbacks.
-
It does not support recursive monitoring. For example, if I monitor directory A, I can capture the event of creating directory B under A. But we cannot capture events under directory B unless we also add directory B to the watch list.
-
There are few available inotify libraries for Python.
For the first drawback, the common solution is to implement recursive monitoring ourselves. When there is a file/directory creation event in the main directory, we add the corresponding file/directory to the watch list.
But this brings a new problem. If we do it this way for a very large project, the resulting memory consumption is alarming. So, when inotify was designed, some kernel parameters were set to limit this.
The two most common ones are:
-
/proc/sys/fs/inotify/max_queued_events: Limits the length of the event queue. Once events accumulate, new events will be discarded.
-
/proc/sys/fs/inotify/max_user_watches: Limits the number of watchers that can be created per User ID to prevent memory explosion caused by excessive monitoring.
By default, the value of max_user_watches depends on the Linux distribution. For most distributions, its value is relatively small. In other words, once the limit is reached, it will not be possible to add new watchers. This is also the reason why the IDE displays the warning External file changes sync may be slow: The current inotify(7) watch limit is too low.
You can modify the corresponding parameters by editing /etc/sysctl.conf to solve this problem.
Finally#
Linux is truly a treasure trove. I feel like I encounter something new every now and then. So, I decided to record it as a short article and also for my own reference.