nohang

nohang package provides a highly configurable daemon for Linux which is able to correctly prevent out of memory (OOM) and keep system responsiveness in low memory conditions.

The package also includes additional diagnostic tools (oom-sort, psi2log, psi-top).

What is the problem?

OOM conditions may cause freezes, livelocks, drop caches and processes to be killed (via sending SIGKILL) instead of trying to terminate them correctly (via sending SIGTERM or takes other corrective action). Some applications may crash if it's impossible to allocate memory.

Here are the statements of some users:

"How do I prevent Linux from freezing when out of memory? Today I (accidentally) ran some program on my Linux box that quickly used a lot of memory. My system froze, became unresponsive and thus I was unable to kill the offender. How can I prevent this in the future? Can't it at least keep a responsive core or something running?"

— serverfault

"With or without swap it still freezes before the OOM killer gets run automatically. This is really a kernel bug that should be fixed (i.e. run OOM killer earlier, before dropping all disk cache). Unfortunately kernel developers and a lot of other folk fail to see the problem. Common suggestions such as disable/enable swap, buy more RAM, run less processes, set limits etc. do not address the underlying problem that the kernel's low memory handling sucks camel's balls."

— serverfault

"The traditional Linux OOM killer works fine in some cases, but in others it kicks in too late, resulting in the system entering a livelock for an indeterminate period."

— engineering.fb.com

Also look at these discussions:

• Why are low memory conditions handled so badly? [r/linux]
• Memory management "more effective" on Windows than Linux? (in preventing total system lockup) [r/linux]
• Let's talk about the elephant in the room - the Linux kernel's inability to gracefully handle low memory pressure [original LKML post | r/linux | Hacker News | slashdot | phoronix | opennet.ru | linux.org.ru]

Solution

Use one of the userspace OOM killers:

• earlyoom: This is a simple, stable and tiny OOM prevention daemon written in C (the best choice for emedded and old servers). It has a minimum dependencies and can work with oldest kernels. It is enabled by default on Fedora 32 Workstation (and F33 KDE).
• oomd: This is a userspace OOM killer for linux systems written in C++ and developed by Facebook. This is the best choice for use in large data centers. It needs Linux 4.20+.
• systemd-oomd: Provided by systemd as systemd-oomd.service that uses cgroups-v2 and pressure stall information (PSI) to monitor and take action on processes before an OOM occurs in kernel space. It's used by default on desktop versions of Fedora 34.
• low-memory-monitor: There's a project announcement.
• psi-monitor: It's used by default on Endless OS.
• nohang: nohang is earlyoom on steroids and has many useful features, see below. Maybe this is a good choice for modern desktops and servers if you need fine-tuning. Previously it was used by default on Garuda Linux.

Use these tools to improve responsiveness during heavy swapping:

• MGLRU patchset is merged in Linux 6.1. Setting min_ttl_ms > 50 can help you.
• le9-patch: [PATCH] mm: Protect clean file pages under memory pressure to prevent thrashing, avoid high latency and prevent livelock in near-OOM conditions. It's kernel-side solution that can fix the OOM killer behavior.
• prelockd: Lock executables and shared libraries in memory to improve system responsiveness under low-memory conditions.
• memavaild: Keep amount of available memory by evicting memory of selected cgroups into swap space.
• uresourced: This daemon will give resource allocations to active graphical users. It's enabled by default on Fedora 33 Workstation.

Of course, you can also download more RAM, tune virtual memory, use zram/zswap and use limits for cgroups.

Features

• Sending the SIGTERM signal is default corrective action. If the victim does not respond to SIGTERM, with a further drop in the level of memory it gets SIGKILL;
• Customizing victim selection: impact on the badness of processes via matching their names, cgroups, exe realpathes, environs, cmdlines and euids with specified regular expressions;
• Customizing corrective actions: if the name or control group of the victim matches a certain regex pattern, you can run any command instead of sending the SIGTERM signal (the default corrective action) to the victim. For example:
  • systemctl restart foo;
  • kill -INT $PID (you can override the signal sent to the victim, $PID will be replaced by the victim's PID).
• GUI notifications:
  • Notification of corrective actions taken and displaying the name and PID of the victim;
  • Low memory warnings.
• zram support (mem_used_total as a trigger);
• PSI (pressure stall information) support;
• Optional checking kernel messages for OOM events;
• Easy setup with configuration files (nohang.conf, nohang-desktop.conf).

Demo

nohang prevents Out Of Memory with GUI notifications:

• https://youtu.be/ChTNu9m7uMU – just old demo without swap space.
• https://youtu.be/UCwZS5uNLu0 – running multiple fast memory hogs at the same time without swap space.
• https://youtu.be/PLVWgNrVNlc – opening multiple chromium tabs with 2.3 GiB memory and 1.8 GiB swap space on zram.

Requirements

For basic usage:

• Linux (>= 3.14, since MemAvailable appeared in /proc/meminfo)
• Python (>= 3.3)

To respond to PSI metrics (optional):

• Linux (>= 4.20) with CONFIG_PSI=y

To show GUI notifications (optional):

• notification server (most of desktop environments use their own implementations)
• libnotify (Arch Linux, Fedora, openSUSE) or libnotify-bin (Debian GNU/Linux, Ubuntu)
• sudo if nohang started with UID=0.

Memory and CPU usage

• VmRSS is about 10–14 MiB instead of the settings, about 10–11 MiB by default (with Python <= 3.8), about 16–17 MiB with Python 3.9.
• CPU usage depends on the level of available memory and monitoring intensity.

Warnings

• the daemon runs with super-user privileges and has full access to all private memory of all processes and sensitive user data;
• the daemon does not forbid you to shoot yourself in the foot: with some settings, unwanted killings of processes can occur;
• the daemon is not a panacea: there are no universal settings that reliably protect against all types of threats.

Known problems

• The documentation is terrible.
• The ZFS ARC cache is memory-reclaimable, like the Linux buffer cache. However, in contrast to the buffer cache, it currently does not count to MemAvailable (see openzfs/zfs#10255). See also https://github.com/rfjakob/earlyoom/pull/191 and https://github.com/hakavlad/nohang/issues