I came across this problem where every few days one of my Java servers is dying. To find out why, I ran
cat /var/log/syslog which produces:
Sep 21 06:36:24 ip-172-31-42-5 systemd-timesyncd(16235): Network configuration changed, trying to establish connection. Sep 21 06:36:24 ip-172-31-42-5 systemd-timesyncd(16235): Synchronized to time server 188.8.131.52:123 (ntp.ubuntu.com). Sep 21 06:46:45 ip-172-31-42-5 systemd(1): Starting Daily apt upgrade and clean activities... Sep 21 06:47:35 ip-172-31-42-5 kernel: (5731409.370359) systemd-resolve invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 ...(skipping a bunch) ... Sep 21 06:47:35 ip-172-31-42-5 kernel: (5731409.370580) Out of memory: Kill process 28251 (java) score 368 or sacrifice child Sep 21 06:47:35 ip-172-31-42-5 kernel: (5731409.374244) Killed process 28251 (java) total-vm:2613628kB, anon-rss:178048kB, file-rss:0kB, shmem-rss:0kB Sep 21 06:47:35 ip-172-31-42-5 kernel: (5731409.403233) oom_reaper: reaped process 28251 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
After reading this post, I learned that the oom_reaper stops the largest memory-intensive process when any process tries to allocate too much memory. Well, I suspect that systemd-resolve trying to request a ridiculous amount of memory, and then the reaper stops my bad java process.
I could handle that by putting it down
vm.oom_kill_allocating_task As suggested in the link above, I'm trying to figure out why systemd-resolve is trying to request such an absurd amount of memory. Does anyone know what the basic problem is here?
Edit: When the Java server is running, memory usage is only 234M / 460M