linux – OOM killer invoked gfp_mask=0x24201ca

I am trying to find out what causes memory to overload OOM killer to perform it actions. It is coming once in around three hours, average live client count is around 300, and I cannot figure out what is the trigger in this cause, because oom-killer can be called from different processes starting from server-host process, to beamium and noderig, the server-host process also fills up the memory even tho clients might disconnect from the server/the count of them is lowering. Looking for extra knowledge because I am just out of ideas, any way to debug deeper? Maybe the problem is around swap? free -h information

System log:

Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:366872)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:502088)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:502080)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:501048)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:501040)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:501032)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:501024)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:502104)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:502096)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:367496)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:336808)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:336792)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:336784)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:336768)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:848456)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:845352)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:464)
Mar 31 00:26:47 pashamachine server(5514): (N) IPaddress:62588 has been disconnected.
Mar 31 00:26:58 pashamachine server(5514): (N) IPaddress:60275 has been disconnected.
Mar 31 00:27:11 pashamachine kernel: mysqld invoked oom-killer: gfp_mask=0x24201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=0, order=0, oom_score_adj=0
Mar 31 00:27:11 pashamachine kernel: mysqld cpuset=/ mems_allowed=0
Mar 31 00:27:11 pashamachine kernel: CPU: 4 PID: 2517 Comm: mysqld Not tainted 4.9.168-xxxx-std-ipv6-64 #665790
Mar 31 00:27:11 pashamachine kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./E3C242D4U2-2T, BIOS L0.09E 03/14/2019
Mar 31 00:27:11 pashamachine kernel: ffffb5138692b9e0 ffffffffaf679ef7 ffffb5138692bb98 ffff9f160a8cc380
Mar 31 00:27:11 pashamachine kernel: ffffb5138692ba58 ffffffffaf22fd97 0000000000000000 0000000000000000
Mar 31 00:27:11 pashamachine kernel: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
Mar 31 00:27:11 pashamachine kernel: Call Trace:
Mar 31 00:27:11 pashamachine kernel: (<ffffffffaf679ef7>) dump_stack+0x4d/0x66
Mar 31 00:27:11 pashamachine kernel: (<ffffffffaf22fd97>) dump_header+0x76/0x1f1
Mar 31 00:27:11 pashamachine kernel: (<ffffffffaf1bf37e>) oom_kill_process+0x20e/0x3e0
Mar 31 00:27:11 pashamachine kernel: (<ffffffffaf1bf87b>) out_of_memory+0x11b/0x4a0
Mar 31 00:27:11 pashamachine kernel: (<ffffffffaf1c3f44>) __alloc_pages_slowpath+0x994/0xb80
Mar 31 00:27:11 pashamachine kernel: (<ffffffffaf1c42e7>) __alloc_pages_nodemask+0x147/0x1d0
Mar 31 00:27:11 pashamachine kernel: (<ffffffffaf20d59e>) alloc_pages_current+0x9e/0x150
Mar 31 00:27:11 pashamachine kernel: (<ffffffffaf1bad93>) __page_cache_alloc+0xa3/0xe0
Mar 31 00:27:11 pashamachine kernel: (<ffffffffaf1bbc38>) ? pagecache_get_page+0x28/0x220
Mar 31 00:27:11 pashamachine kernel: (<ffffffffaf1bc12d>) filemap_fault+0x2fd/0x4a0
Mar 31 00:27:11 pashamachine kernel: (<ffffffffaf30cec1>) ext4_filemap_fault+0x31/0x50
Mar 31 00:27:11 pashamachine kernel: (<ffffffffaf1e7a53>) __do_fault+0xa3/0x1a0
Mar 31 00:27:11 pashamachine kernel: (<ffffffffaf1eb43f>) handle_mm_fault+0xb6f/0x1120
Mar 31 00:27:11 pashamachine kernel: (<ffffffffaf09ea4d>) __do_page_fault+0x22d/0x450
Mar 31 00:27:11 pashamachine kernel: (<ffffffffaf09ecac>) do_page_fault+0xc/0x10
Mar 31 00:27:11 pashamachine kernel: (<ffffffffafeac282>) page_fault+0x22/0x30
Mar 31 00:27:11 pashamachine kernel: Mem-Info:
Mar 31 00:27:11 pashamachine kernel: active_anon:7602864 inactive_anon:447235 isolated_anon:0
Mar 31 00:27:11 pashamachine kernel: active_file:162 inactive_file:347 isolated_file:0
Mar 31 00:27:11 pashamachine kernel: unevictable:1 dirty:0 writeback:0 unstable:0
Mar 31 00:27:11 pashamachine kernel: slab_reclaimable:5973 slab_unreclaimable:19315
Mar 31 00:27:11 pashamachine kernel: mapped:964 shmem:79951 pagetables:17086 bounce:0
Mar 31 00:27:11 pashamachine kernel: free:50740 free_pcp:387 free_cma:0
Mar 31 00:27:11 pashamachine kernel: Node 0 active_anon:30411456kB inactive_anon:1788940kB active_file:728kB inactive_file:680kB unevictable:4kB isolated(anon):0kB isolated(file):0kB mapped:4028kB dirty:0kB writeback:0kB shmem:319804kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 9269248kB writeback_tmp:0kB unstable:0kB pages_scanned:178 all_unreclaimable? no
Mar 31 00:27:11 pashamachine kernel: Node 0 DMA free:15896kB min:32kB low:44kB high:56kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15896kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Mar 31 00:27:11 pashamachine kernel: lowmem_reserve(): 0 2017 31924 31924
Mar 31 00:27:11 pashamachine kernel: Node 0 DMA32 free:123888kB min:4268kB low:6332kB high:8396kB active_anon:1767176kB inactive_anon:241356kB active_file:220kB inactive_file:600kB unevictable:0kB writepending:0kB present:2140472kB managed:2140468kB mlocked:0kB slab_reclaimable:252kB slab_unreclaimable:208kB kernel_stack:64kB pagetables:3928kB bounce:0kB free_pcp:764kB local_pcp:0kB free_cma:0kB
Mar 31 00:27:11 pashamachine kernel: lowmem_reserve(): 0 0 29907 29907
Mar 31 00:27:11 pashamachine kernel: Node 0 Normal free:63224kB min:63280kB low:93904kB high:124528kB active_anon:28644280kB inactive_anon:1547584kB active_file:360kB inactive_file:524kB unevictable:4kB writepending:0kB present:31178752kB managed:30628620kB mlocked:4kB slab_reclaimable:23640kB slab_unreclaimable:77052kB kernel_stack:7088kB pagetables:64416kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Mar 31 00:27:11 pashamachine kernel: lowmem_reserve(): 0 0 0 0
Mar 31 00:27:11 pashamachine kernel: Node 0 DMA: 2*4kB (U) 2*8kB (U) 0*16kB 2*32kB (U) 3*64kB (U) 2*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15896kB
Mar 31 00:27:11 pashamachine kernel: Node 0 DMA32: 14*4kB (UME) 50*8kB (UM) 88*16kB (UME) 124*32kB (UME) 68*64kB (UME) 48*128kB (UME) 19*256kB (UE) 12*512kB (UE) 7*1024kB (UE) 2*2048kB (UM) 21*4096kB (UMH) = 124616kB
Mar 31 00:27:11 pashamachine kernel: Node 0 Normal: 473*4kB (UME) 261*8kB (UMEH) 269*16kB (UMEH) 234*32kB (UMEH) 170*64kB (UMEH) 81*128kB (UMEH) 53*256kB (UMEH) 23*512kB (UME) 1*1024kB (H) 0*2048kB 0*4096kB = 63388kB
Mar 31 00:27:11 pashamachine kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Mar 31 00:27:11 pashamachine kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Mar 31 00:27:11 pashamachine kernel: 231789 total pagecache pages
Mar 31 00:27:11 pashamachine kernel: 151263 pages in swap cache
Mar 31 00:27:11 pashamachine kernel: Swap cache stats: add 839809, delete 688546, find 3766057/3789646
Mar 31 00:27:11 pashamachine kernel: Free swap  = 0kB
Mar 31 00:27:11 pashamachine kernel: Total swap = 1046520kB
Mar 31 00:27:11 pashamachine kernel: 8333804 pages RAM
Mar 31 00:27:11 pashamachine kernel: 0 pages HighMem/MovableOnly
Mar 31 00:27:11 pashamachine kernel: 137558 pages reserved
Mar 31 00:27:11 pashamachine kernel: 0 pages hwpoisoned
Mar 31 00:27:11 pashamachine kernel: ( pid )   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Mar 31 00:27:11 pashamachine kernel: (  480)     0   480     9091      900      15       3       79             0 systemd-journal
Mar 31 00:27:11 pashamachine kernel: (  508)     0   508     4697       14       9       3      289         -1000 systemd-udevd
Mar 31 00:27:11 pashamachine kernel: (  513)   101   513     3563       57      10       3      105             0 systemd-network
Mar 31 00:27:11 pashamachine kernel: (  681)     0   681      810       62       5       3       28             0 mdadm
Mar 31 00:27:11 pashamachine kernel: (  743)   100   743    21772       29      12       3       99             0 systemd-timesyn
Mar 31 00:27:11 pashamachine kernel: (  773)   106   773   255875     8927      77       4     4618             0 named
Mar 31 00:27:11 pashamachine kernel: (  780)     0   780     2561       65       8       3       63             0 irqbalance
Mar 31 00:27:11 pashamachine kernel: (  798)     0   798     2437       25       8       3       42             0 cron
Mar 31 00:27:11 pashamachine kernel: (  802)     0   802     2964       39       9       3      105             0 systemd-logind
Mar 31 00:27:11 pashamachine kernel: (  806)   107   806     2183      120       7       3       44          -900 dbus-daemon
Mar 31 00:27:11 pashamachine kernel: (  812)   108   812    23477     2583      50       4    10441             0 beamium
Mar 31 00:27:11 pashamachine kernel: (  837)     0   837     6834       32      16       3      157         -1000 sshd
Mar 31 00:27:11 pashamachine kernel: (  842)     0   842   498754    22631     120       7     4345             0 noderig
Mar 31 00:27:11 pashamachine kernel: (  846)     0   846     1656        0       7       3       30             0 agetty
Mar 31 00:27:11 pashamachine kernel: (  848)     0   848     1575        0       7       3      116             0 login
Mar 31 00:27:11 pashamachine kernel: (  850)     0   850     1656        0       8       3       30             0 agetty
Mar 31 00:27:11 pashamachine kernel: ( 1679)     0  1679     2013        1       6       3      145             0 screen
Mar 31 00:27:11 pashamachine kernel: ( 1680)     0  1680     2059        1       7       3      135             0 bash
Mar 31 00:27:11 pashamachine kernel: ( 2296)     0  2296     2059        1       7       3      144             0 bash
Mar 31 00:27:11 pashamachine kernel: (30306)     0 30306     2015        1       7       3      149             0 screen
Mar 31 00:27:11 pashamachine kernel: (30307)     0 30307     2059        1       7       3      137             0 bash
Mar 31 00:27:11 pashamachine kernel: (30308)   109 30308  2983591  1376774    3615      16   237632             0 mysqld
Mar 31 00:27:11 pashamachine kernel: ( 6887)     0  6887    57103      401      16       4       83             0 rsyslogd
Mar 31 00:27:11 pashamachine kernel: ( 5514)     0  5514  8497142  6404668   12981      48        0             0 server
Mar 31 00:27:11 pashamachine kernel: ( 7436)     0  7436     6954      229      17       3        0             0 sshd
Mar 31 00:27:11 pashamachine kernel: ( 7443)     0  7443      608       24       5       3        0             0 sftp-server
Mar 31 00:27:11 pashamachine kernel: ( 7753)     0  7753     6955      253      17       3        0             0 sshd
Mar 31 00:27:11 pashamachine kernel: ( 7759)     0  7759     2059      140       7       3        0             0 bash
Mar 31 00:27:11 pashamachine kernel: ( 8685)     0  8685     3769      206      11       3        0             0 top
Mar 31 00:27:11 pashamachine kernel: Out of memory: Kill process 5514 (server) score 736 or sacrifice child
Mar 31 00:27:11 pashamachine kernel: Killed process 5514 (server) total-vm:33988568kB, anon-rss:25618672kB, file-rss:0kB, shmem-rss:0kB