I am trying to find out what causes memory to overload OOM killer to perform it actions. It is coming once in around three hours, average live client count is around 300, and I cannot figure out what is the trigger in this cause, because oom-killer can be called from different processes starting from server-host process, to beamium and noderig, the server-host process also fills up the memory even tho clients might disconnect from the server/the count of them is lowering. Looking for extra knowledge because I am just out of ideas, any way to debug deeper? Maybe the problem is around swap? free -h information
System log:
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:366872)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:502088)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:502080)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:501048)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:501040)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:501032)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:501024)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:502104)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:502096)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:367496)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:336808)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:336792)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:336784)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:336768)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:848456)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:845352)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:464)
Mar 31 00:26:47 pashamachine server(5514): (N) IPaddress:62588 has been disconnected.
Mar 31 00:26:58 pashamachine server(5514): (N) IPaddress:60275 has been disconnected.
Mar 31 00:27:11 pashamachine kernel: mysqld invoked oom-killer: gfp_mask=0x24201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=0, order=0, oom_score_adj=0
Mar 31 00:27:11 pashamachine kernel: mysqld cpuset=/ mems_allowed=0
Mar 31 00:27:11 pashamachine kernel: CPU: 4 PID: 2517 Comm: mysqld Not tainted 4.9.168-xxxx-std-ipv6-64 #665790
Mar 31 00:27:11 pashamachine kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./E3C242D4U2-2T, BIOS L0.09E 03/14/2019
Mar 31 00:27:11 pashamachine kernel: ffffb5138692b9e0 ffffffffaf679ef7 ffffb5138692bb98 ffff9f160a8cc380
Mar 31 00:27:11 pashamachine kernel: ffffb5138692ba58 ffffffffaf22fd97 0000000000000000 0000000000000000
Mar 31 00:27:11 pashamachine kernel: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
Mar 31 00:27:11 pashamachine kernel: Call Trace:
Mar 31 00:27:11 pashamachine kernel: (<ffffffffaf679ef7>) dump_stack+0x4d/0x66
Mar 31 00:27:11 pashamachine kernel: (<ffffffffaf22fd97>) dump_header+0x76/0x1f1
Mar 31 00:27:11 pashamachine kernel: (<ffffffffaf1bf37e>) oom_kill_process+0x20e/0x3e0
Mar 31 00:27:11 pashamachine kernel: (<ffffffffaf1bf87b>) out_of_memory+0x11b/0x4a0
Mar 31 00:27:11 pashamachine kernel: (<ffffffffaf1c3f44>) __alloc_pages_slowpath+0x994/0xb80
Mar 31 00:27:11 pashamachine kernel: (<ffffffffaf1c42e7>) __alloc_pages_nodemask+0x147/0x1d0
Mar 31 00:27:11 pashamachine kernel: (<ffffffffaf20d59e>) alloc_pages_current+0x9e/0x150
Mar 31 00:27:11 pashamachine kernel: (<ffffffffaf1bad93>) __page_cache_alloc+0xa3/0xe0
Mar 31 00:27:11 pashamachine kernel: (<ffffffffaf1bbc38>) ? pagecache_get_page+0x28/0x220
Mar 31 00:27:11 pashamachine kernel: (<ffffffffaf1bc12d>) filemap_fault+0x2fd/0x4a0
Mar 31 00:27:11 pashamachine kernel: (<ffffffffaf30cec1>) ext4_filemap_fault+0x31/0x50
Mar 31 00:27:11 pashamachine kernel: (<ffffffffaf1e7a53>) __do_fault+0xa3/0x1a0
Mar 31 00:27:11 pashamachine kernel: (<ffffffffaf1eb43f>) handle_mm_fault+0xb6f/0x1120
Mar 31 00:27:11 pashamachine kernel: (<ffffffffaf09ea4d>) __do_page_fault+0x22d/0x450
Mar 31 00:27:11 pashamachine kernel: (<ffffffffaf09ecac>) do_page_fault+0xc/0x10
Mar 31 00:27:11 pashamachine kernel: (<ffffffffafeac282>) page_fault+0x22/0x30
Mar 31 00:27:11 pashamachine kernel: Mem-Info:
Mar 31 00:27:11 pashamachine kernel: active_anon:7602864 inactive_anon:447235 isolated_anon:0
Mar 31 00:27:11 pashamachine kernel: active_file:162 inactive_file:347 isolated_file:0
Mar 31 00:27:11 pashamachine kernel: unevictable:1 dirty:0 writeback:0 unstable:0
Mar 31 00:27:11 pashamachine kernel: slab_reclaimable:5973 slab_unreclaimable:19315
Mar 31 00:27:11 pashamachine kernel: mapped:964 shmem:79951 pagetables:17086 bounce:0
Mar 31 00:27:11 pashamachine kernel: free:50740 free_pcp:387 free_cma:0
Mar 31 00:27:11 pashamachine kernel: Node 0 active_anon:30411456kB inactive_anon:1788940kB active_file:728kB inactive_file:680kB unevictable:4kB isolated(anon):0kB isolated(file):0kB mapped:4028kB dirty:0kB writeback:0kB shmem:319804kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 9269248kB writeback_tmp:0kB unstable:0kB pages_scanned:178 all_unreclaimable? no
Mar 31 00:27:11 pashamachine kernel: Node 0 DMA free:15896kB min:32kB low:44kB high:56kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15896kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Mar 31 00:27:11 pashamachine kernel: lowmem_reserve(): 0 2017 31924 31924
Mar 31 00:27:11 pashamachine kernel: Node 0 DMA32 free:123888kB min:4268kB low:6332kB high:8396kB active_anon:1767176kB inactive_anon:241356kB active_file:220kB inactive_file:600kB unevictable:0kB writepending:0kB present:2140472kB managed:2140468kB mlocked:0kB slab_reclaimable:252kB slab_unreclaimable:208kB kernel_stack:64kB pagetables:3928kB bounce:0kB free_pcp:764kB local_pcp:0kB free_cma:0kB
Mar 31 00:27:11 pashamachine kernel: lowmem_reserve(): 0 0 29907 29907
Mar 31 00:27:11 pashamachine kernel: Node 0 Normal free:63224kB min:63280kB low:93904kB high:124528kB active_anon:28644280kB inactive_anon:1547584kB active_file:360kB inactive_file:524kB unevictable:4kB writepending:0kB present:31178752kB managed:30628620kB mlocked:4kB slab_reclaimable:23640kB slab_unreclaimable:77052kB kernel_stack:7088kB pagetables:64416kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Mar 31 00:27:11 pashamachine kernel: lowmem_reserve(): 0 0 0 0
Mar 31 00:27:11 pashamachine kernel: Node 0 DMA: 2*4kB (U) 2*8kB (U) 0*16kB 2*32kB (U) 3*64kB (U) 2*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15896kB
Mar 31 00:27:11 pashamachine kernel: Node 0 DMA32: 14*4kB (UME) 50*8kB (UM) 88*16kB (UME) 124*32kB (UME) 68*64kB (UME) 48*128kB (UME) 19*256kB (UE) 12*512kB (UE) 7*1024kB (UE) 2*2048kB (UM) 21*4096kB (UMH) = 124616kB
Mar 31 00:27:11 pashamachine kernel: Node 0 Normal: 473*4kB (UME) 261*8kB (UMEH) 269*16kB (UMEH) 234*32kB (UMEH) 170*64kB (UMEH) 81*128kB (UMEH) 53*256kB (UMEH) 23*512kB (UME) 1*1024kB (H) 0*2048kB 0*4096kB = 63388kB
Mar 31 00:27:11 pashamachine kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Mar 31 00:27:11 pashamachine kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Mar 31 00:27:11 pashamachine kernel: 231789 total pagecache pages
Mar 31 00:27:11 pashamachine kernel: 151263 pages in swap cache
Mar 31 00:27:11 pashamachine kernel: Swap cache stats: add 839809, delete 688546, find 3766057/3789646
Mar 31 00:27:11 pashamachine kernel: Free swap = 0kB
Mar 31 00:27:11 pashamachine kernel: Total swap = 1046520kB
Mar 31 00:27:11 pashamachine kernel: 8333804 pages RAM
Mar 31 00:27:11 pashamachine kernel: 0 pages HighMem/MovableOnly
Mar 31 00:27:11 pashamachine kernel: 137558 pages reserved
Mar 31 00:27:11 pashamachine kernel: 0 pages hwpoisoned
Mar 31 00:27:11 pashamachine kernel: ( pid ) uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
Mar 31 00:27:11 pashamachine kernel: ( 480) 0 480 9091 900 15 3 79 0 systemd-journal
Mar 31 00:27:11 pashamachine kernel: ( 508) 0 508 4697 14 9 3 289 -1000 systemd-udevd
Mar 31 00:27:11 pashamachine kernel: ( 513) 101 513 3563 57 10 3 105 0 systemd-network
Mar 31 00:27:11 pashamachine kernel: ( 681) 0 681 810 62 5 3 28 0 mdadm
Mar 31 00:27:11 pashamachine kernel: ( 743) 100 743 21772 29 12 3 99 0 systemd-timesyn
Mar 31 00:27:11 pashamachine kernel: ( 773) 106 773 255875 8927 77 4 4618 0 named
Mar 31 00:27:11 pashamachine kernel: ( 780) 0 780 2561 65 8 3 63 0 irqbalance
Mar 31 00:27:11 pashamachine kernel: ( 798) 0 798 2437 25 8 3 42 0 cron
Mar 31 00:27:11 pashamachine kernel: ( 802) 0 802 2964 39 9 3 105 0 systemd-logind
Mar 31 00:27:11 pashamachine kernel: ( 806) 107 806 2183 120 7 3 44 -900 dbus-daemon
Mar 31 00:27:11 pashamachine kernel: ( 812) 108 812 23477 2583 50 4 10441 0 beamium
Mar 31 00:27:11 pashamachine kernel: ( 837) 0 837 6834 32 16 3 157 -1000 sshd
Mar 31 00:27:11 pashamachine kernel: ( 842) 0 842 498754 22631 120 7 4345 0 noderig
Mar 31 00:27:11 pashamachine kernel: ( 846) 0 846 1656 0 7 3 30 0 agetty
Mar 31 00:27:11 pashamachine kernel: ( 848) 0 848 1575 0 7 3 116 0 login
Mar 31 00:27:11 pashamachine kernel: ( 850) 0 850 1656 0 8 3 30 0 agetty
Mar 31 00:27:11 pashamachine kernel: ( 1679) 0 1679 2013 1 6 3 145 0 screen
Mar 31 00:27:11 pashamachine kernel: ( 1680) 0 1680 2059 1 7 3 135 0 bash
Mar 31 00:27:11 pashamachine kernel: ( 2296) 0 2296 2059 1 7 3 144 0 bash
Mar 31 00:27:11 pashamachine kernel: (30306) 0 30306 2015 1 7 3 149 0 screen
Mar 31 00:27:11 pashamachine kernel: (30307) 0 30307 2059 1 7 3 137 0 bash
Mar 31 00:27:11 pashamachine kernel: (30308) 109 30308 2983591 1376774 3615 16 237632 0 mysqld
Mar 31 00:27:11 pashamachine kernel: ( 6887) 0 6887 57103 401 16 4 83 0 rsyslogd
Mar 31 00:27:11 pashamachine kernel: ( 5514) 0 5514 8497142 6404668 12981 48 0 0 server
Mar 31 00:27:11 pashamachine kernel: ( 7436) 0 7436 6954 229 17 3 0 0 sshd
Mar 31 00:27:11 pashamachine kernel: ( 7443) 0 7443 608 24 5 3 0 0 sftp-server
Mar 31 00:27:11 pashamachine kernel: ( 7753) 0 7753 6955 253 17 3 0 0 sshd
Mar 31 00:27:11 pashamachine kernel: ( 7759) 0 7759 2059 140 7 3 0 0 bash
Mar 31 00:27:11 pashamachine kernel: ( 8685) 0 8685 3769 206 11 3 0 0 top
Mar 31 00:27:11 pashamachine kernel: Out of memory: Kill process 5514 (server) score 736 or sacrifice child
Mar 31 00:27:11 pashamachine kernel: Killed process 5514 (server) total-vm:33988568kB, anon-rss:25618672kB, file-rss:0kB, shmem-rss:0kB