mysql – HAProxy has problems with http and tcp checks: Socket error, info: “Connection reset by peer”

i got five machines in the same network, firewalls are disabled:

haproxy 10.0.0.2
mysql-1 10.0.0.3
mysql-2 10.0.0.4
mysql-3 10.0.0.5
mysql-4 10.0.0.6

The problem is, that either http-check or tcp-check in around 2/3 of the checks fails (but sometimes somehow not!).

The very interesting part on this is, that from the haproxy-machine, i can do both checks via telnet without any problems to any of the mysql-servers:

from the haproxy machine:

telnet 10.0.0.2 9200

results in

Trying 10.0.0.2...
Connected to 10.0.0.2.
Escape character is '^)'.
HTTP/1.1 200 OK

Content-Type: text/html

Content-Length: 43



<html><body>MySQL is running.</body></html>



Connection closed by foreign host.

And the tcp check is also okay:

telnet 10.0.0.2 3306

results in:

Trying 10.1.0.4...
Connected to 10.1.0.4.
Escape character is '^)'.
s
5.5.5-10.5.10-MariaDB-1:10.5.10+maria~focal-log<?r_dXk3XC?'5wxvi).Mx5;mysql_native_password

Here is my haproxy.cfg

global
        pidfile /var/run/haproxy.pid
        daemon
        user haproxy
        group haproxy
        stats socket /var/run/haproxy.socket user haproxy group haproxy mode 600 level admin
        node haproxy_10.0.0.2
        description haproxy server

        #* Performance Tuning
        maxconn 8192
        spread-checks 3
        quiet
defaults
        log    global
        mode    tcp
        option  dontlognull
        option tcp-smart-accept
        option tcp-smart-connect
        #option dontlog-normal
        retries 3
        option redispatch
        maxconn 8192
        timeout check   3500ms
        timeout queue   3500ms
        timeout connect 3500ms
        timeout client  10800s
        timeout server  10800s

userlist STATSUSERS
        group admin users admin
        user admin insecure-password admin
        user stats insecure-password admin

listen admin_page
        bind *:9600
        mode http
        stats enable
        stats refresh 60s
        stats uri /
        acl AuthOkay_ReadOnly http_auth(STATSUSERS)
        acl AuthOkay_Admin http_auth_group(STATSUSERS) admin
        stats http-request auth realm admin_page unless AuthOkay_ReadOnly
        #stats admin if AuthOkay_Admin

listen  haproxy_10.0.0.6_3307_rw
        bind *:3307
        mode tcp
        tcp-check connect port 9200
        timeout client  10800s
        timeout server  10800s
        balance leastconn
        option httpchk
#        option allbackups
        default-server port 9200 inter 2s downinter 5s rise 3 fall 2 slowstart 60s maxconn 64 maxqueue 128 weight 100
        server 10.0.0.3 10.1.0.3:3306 check
        server 10.0.0.4 10.1.0.4:3306 check
        server 10.0.0.5 10.1.0.5:3306 check
        server 10.0.0.6 10.1.0.6:3306 check

I get the following error messages:

May 31 22:40:05 HAProxy-Cluster-LoadBalancer haproxy(5648): (WARNING) 150/224005 (5648) : Server haproxy_10.0.0.2_3307_rw/10.0.0.3 is UP, reason: Layer7 check passed, code: 200, info: "OK", check duration: 18ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
May 31 22:40:09 HAProxy-Cluster-LoadBalancer haproxy(5648): (WARNING) 150/224009 (5648) : Server haproxy_10.0.0.2_3307_rw/10.0.0.3 is DOWN, reason: Socket error, info: "Connection reset by peer", check duration: 18ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
May 31 22:40:09 HAProxy-Cluster-LoadBalancer haproxy(5648): (ALERT) 150/224009 (5648) : proxy 'haproxy_10.0.0.2_3307_rw' has no server available!
May 31 22:40:18 HAProxy-Cluster-LoadBalancer haproxy(5648): (WARNING) 150/224018 (5648) : Server haproxy_10.0.0.2_3307_rw/10.0.0.3 is UP, reason: Layer7 check passed, code: 200, info: "OK", check duration: 20ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
May 31 22:40:22 HAProxy-Cluster-LoadBalancer haproxy(5648): (WARNING) 150/224022 (5648) : Server haproxy_10.0.0.2_3307_rw/10.0.0.3 is DOWN, reason: Socket error, info: "Connection reset by peer", check duration: 21ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
May 31 22:40:22 HAProxy-Cluster-LoadBalancer haproxy(5648): (ALERT) 150/224022 (5648) : proxy 'haproxy_10.0.0.2_3307_rw' has no server available!
May 31 22:40:27 HAProxy-Cluster-LoadBalancer haproxy(5648): (WARNING) 150/224027 (5648) : Server haproxy_10.0.0.2_3307_rw/10.0.0.6 is UP, reason: Layer7 check passed, code: 200, info: "OK", check duration: 21ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
May 31 22:40:31 HAProxy-Cluster-LoadBalancer haproxy(5648): (WARNING) 150/224031 (5648) : Server haproxy_10.0.0.2_3307_rw/10.0.0.6 is DOWN, reason: Socket error, info: "Connection reset by peer", check duration: 24ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
May 31 22:40:31 HAProxy-Cluster-LoadBalancer haproxy(5648): (ALERT) 150/224031 (5648) : proxy 'haproxy_10.0.0.2_3307_rw' has no server available!
May 31 22:40:55 HAProxy-Cluster-LoadBalancer haproxy(5648): (WARNING) 150/224055 (5648) : Server haproxy_10.0.0.2_3307_rw/10.0.0.3 is UP, reason: Layer7 check passed, code: 200, info: "OK", check duration: 24ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
May 31 22:41:05 HAProxy-Cluster-LoadBalancer haproxy(5648): (WARNING) 150/224105 (5648) : Server haproxy_10.0.0.2_3307_rw/10.0.0.4 is UP, reason: Layer7 check passed, code: 200, info: "OK", check duration: 33ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
May 31 22:41:17 HAProxy-Cluster-LoadBalancer haproxy(5648): (WARNING) 150/224117 (5648) : Server haproxy_10.0.0.2_3307_rw/10.0.0.3 is DOWN, reason: Socket error, info: "Connection reset by peer", check duration: 19ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
May 31 22:41:22 HAProxy-Cluster-LoadBalancer haproxy(5648): (WARNING) 150/224122 (5648) : Server haproxy_10.0.0.2_3307_rw/10.0.0.6 is UP, reason: Layer7 check passed, code: 200, info: "OK", check duration: 20ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
May 31 22:41:26 HAProxy-Cluster-LoadBalancer haproxy(5648): (WARNING) 150/224126 (5648) : Server haproxy_10.0.0.2_3307_rw/10.0.0.6 is DOWN, reason: Socket error, info: "Connection reset by peer", check duration: 23ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
May 31 22:41:30 HAProxy-Cluster-LoadBalancer haproxy(5648): (WARNING) 150/224130 (5648) : Server haproxy_10.0.0.2_3307_rw/10.0.0.4 is DOWN, reason: Socket error, info: "Connection reset by peer", check duration: 18ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
May 31 22:41:30 HAProxy-Cluster-LoadBalancer haproxy(5648): (ALERT) 150/224130 (5648) : proxy 'haproxy_10.0.0.2_3307_rw' has no server available!

So what the hell is going on? Is there a problem on my MySQL-instances?
Disabling both checks results in no errors, but then i wont need haproxy at all….

Here is a screenshot of the status-page:

screenshot of status page

Sometimes all four mysql servers are “down” at the same time, but they arent at all.

Thank you in advance, Louis