Among other mitigating mechanisms, we’re trying to use iptables to limit the rate of incoming requests to a set number of requests per second. Generally the rate of requests is pretty steady, but under certain circumstances we experience a short burst of a large number of requests. We’re experimenting with using limit and burst iptables rules to address this.
To tune the limit and burst values, I have a test client that can send a precise number of requests per second. To start with, I’m setting the
--burst values as high as possible (10k for each). The rules are set like so:
sudo iptables -t mangle -I INPUT -i <iface> -m limit --limit 10000/s --limit-burst 10000 -p tcp -m multiport --dports 8080:8101,8888 -m tcp --syn -j ACCEPT sudo iptables -t mangle -A INPUT -i <iface> -p tcp -m multiport --dports 8080:8101,8888 -m tcp --syn -j DROP
So I expect that over 10,000 requests per second, after the initial 10,000 requests, the first ACCEPT rule will start not matching and the following DROP rule will take effect and SYNs will not receive their ACKs.
What I’m finding is that even at much lower rates, even 3000 requests per second, the DROP rule is still being matched. After my test client sends 10,000 requests at 3,000 rps, the stats for the above rules look like the following:
$ sudo iptables -t mangle -L INPUT -n -v Chain INPUT (policy ACCEPT 80496 packets, 5714K bytes) pkts bytes target prot opt in out source destination 10000 600K ACCEPT tcp -- <iface> * 0.0.0.0/0 0.0.0.0/0 limit: avg 10000/sec burst 10000 multiport dports 8080:8101,8888 tcp flags:0x17/0x02 3219 193K DROP tcp -- <iface> * 0.0.0.0/0 0.0.0.0/0 multiport dports 8080:8101,8888 tcp flags:0x17/0x02
I’ve verified that the client is behaving correctly with regard to the number of requests and the rate. In a packet capture, if I limit on just SYN packets, the SYNs are spaced out by at least .3 milliseconds (skimming through, all of them are between 350 and 500 microseconds). This makes sense since that will result in just under 3,000 requests per second. Here’s a sample image of the wireshark capture with the Time column set to display time since last displayed packet (it’s a small sample, but it’s representative of the rest of the SYNs):
Given this traffic, I don’t expect any dropped packets for two reasons:
- The 3,000 connections per second is well under the
--limitvalue of 10,000 SYNs per second.
- The total number of connections is 10,000, which is equal to the
--burstvalue. Even if the rate were somehow higher than the limit, I still expect the burst to allow these.
Yet, according to the above iptables statistics (and according to the packet capture showing the client timing out on waiting for the SYNs and retransmitting), 3,219 packets got through the very permissive limit and burst settings and got dropped by the second rule.
In case it’s relevant, I’m on a RHEL 7 system.
What am I misunderstanding?