Here comes the garbage API porter again. Today, due to a hidden configuration parameter of Supervisor, a critical project crashed online. I think it's necessary to share this, so I wrote a garbage article.
Cause#
While I was writing code, I suddenly received a bunch of alarm emails, which made me feel that the world is not so lovely.
Then I took a closer look at the exception information. What the hell? A new connection and the infamous [Errno 24] Too many open files? What the hell is going on?
Okay, let's debug.
Bug Investigation#
First of all, as we all know, everything is a file in Linux. So the process of operating network connections is actually about operating File Descriptors. Well, since it's Too many open files, let's first consider whether the system's threshold is too small. So I ran ulimit -a.
Huh? The number in the open files column is not small, it's enough. Then what the hell is this?
Okay, let's check the network connections. I ran netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}' to count the number of connections in various states at that time.
Huh? Interesting, there are too many TIME_WAIT connections. Huh? Interesting, let's bring out my half-hearted skills in kernel network parameters and make some modifications.
# Parameter optimization
# This parameter determines the time that sockets in the FIN-WAIT-2 state will remain open if they are closed by the local end
net.ipv4.tcp_fin_timeout = 30
# This parameter determines the frequency at which TCP sends keepalive messages when keepalive is enabled. The default is 2 hours, changed to 300 seconds
net.ipv4.tcp_keepalive_time = 300
# This parameter enables SYN Cookies. When the SYN wait queue overflows, cookies are used to handle it and can prevent a small number of SYN attacks. The default is 0, which means disabled
net.ipv4.tcp_syncookies = 1
# This parameter determines the time that sockets in the FIN-WAIT-2 state will remain open if they are closed by the local end
net.ipv4.tcp_tw_reuse = 1
# This parameter enables fast recycling of TIME-WAIT sockets in TCP connections. The default is 0, which means disabled. Since there are too many TIME-WAIT connections, we enable fast recycling
net.ipv4.tcp_tw_recycle = 1
net.ipv4.ip_local_port_range = 5000 65000
Add the above configuration items to /etc/sysctl.conf. Then run sysctl -p to take effect. Let's see the results.
Huh! The number of errors is decreasing, and the number of TIME_WAIT connections is gradually becoming normal.
Huh? Wait a minute? The other machines don't have the problem of excessive TIME_WAIT connections. What the hell is this? And fast recycling of TIME_WAIT connections will also bring other side effects (mentioned in the next chapter).
Okay, well, now I suspect that it's a problem with Supervisor. Alright, let's have a look at the documentation.
Hmm, I searched for a long time and found the reason. It's related to a parameter called minfds.
Here's the description:
The minimum number of file descriptors that must be available before supervisord will start successfully. A call to setrlimit will be made to attempt to raise the soft and hard limits of the supervisord process to satisfy minfds. The hard limit may only be raised if supervisord is run as root. supervisord uses file descriptors liberally, and will enter a failure mode when one cannot be obtained from the OS, so it’s useful to be able to specify a minimum value to ensure it doesn’t run out of them during execution. These limits will be inherited by the managed subprocesses. This option is particularly useful on Solaris, which has a low per-process fd limit by default.
In summary, when Supervisor starts, it will determine whether there are enough available file descriptors in the system based on the value of minfds. At the same time, because the services running in Supervisor are forked by Supervisord, due to the parent-child relationship and for security reasons, the maximum number of descriptors allowed for a single process must not exceed the value set by minfds, which is 1024 by default. And then, it added a knife, if you run it as the root user, we will give you the maximum number of descriptors by default!
Damn... So that's the reason. Why would you run services as root for no reason? It's a disaster...
Okay, let's change the parameter, change the parameter.
Follow-up#
Finally, this matter ended like this. Stepped on a landmine and reviewed the kernel's network parameters. Although it feels good, I feel like you, dear Supervisor, are taking the wrong medicine!