Residential Proxy Networks and Kernel-Level Bot Mitigation: nftables Rate-Limiting at the Host Edge
The Problem
IP reputation blocking has a structural blind spot: it assumes that malicious traffic originates from identifiable bad infrastructure. BrightData (formerly Luminati Networks), Oxylabs, and a dozen smaller residential proxy operators have built businesses that eliminate that assumption entirely. Their networks consist of 30 to 100 million residential IP addresses — real consumer broadband connections on Comcast, AT&T, BT, Deutsche Telekom, and every other ISP worldwide. Each IP passes every standard reputation check: it resolves to a real residential ISP, has no datacenter ASN signature, appears in no blocklist, and belongs to a real homeowner who is probably unaware their connection is being used. Blocking the IP would block the legitimate subscriber.
The sourcing mechanisms are deliberate and varied. BrightData’s SDK is embedded in hundreds of free VPN applications and browser extensions: install “FreeVPN Pro,” and buried in the EULA is consent to allow the operator to route traffic through your connection. Oxylabs and similar operators pay mobile application developers to embed their SDKs, which use the device’s residential IP as a proxy exit node whenever the app is running and the device is idle. At the lower end of the market — and for outright malicious use — compromised home routers and IoT devices (primarily cameras, NVRs, and consumer routers running vulnerable firmware) are recruited involuntarily into botnets that serve as residential proxy infrastructure. MikroTik and TP-Link routers with default credentials or unpatched CVEs make up a meaningful fraction of some IoT botnet residential pools.
The consequence for defenders is that standard IP-based controls — fail2ban, per-IP rate limits, MaxMind GeoIP2 datacenter ranges, Spamhaus DROP lists, Cloudflare’s IP reputation engine — all fail against residential proxy traffic. A credential stuffing campaign using a residential proxy network sends each login attempt from a different residential IP. Per-IP rate limiting never triggers. IP reputation shows clean on every IP. The requests are spread across hundreds of ISPs in dozens of countries, making geographic blocking useless unless you are willing to block entire countries of legitimate customers.
The attack surface is concrete:
- Credential stuffing at scale: ATO (account takeover) operations using residential proxies against login endpoints. Tools like SentryMBA, OpenBullet, and Snipr are explicitly designed to integrate with residential proxy APIs. An operator can purchase 1,000 residential IPs for an hour, run 50,000 login attempts, and never reuse an IP. Standard rate limiting of 5 attempts per IP per minute never fires.
- Web scraping bypassing rate limits: Price intelligence scrapers and competitive data harvesters send one request per IP across millions of IPs. No per-IP rate limit is ever reached. The target sees a normal traffic distribution with no anomalous source.
- Shopping bots and ticket scalpers: Each checkout attempt or ticket hold originates from a different IP, defeating per-IP cart reservation limits, CAPTCHA trigger thresholds, and purchase frequency rules.
- IoT botnet volumetric DDoS: Compromised Hikvision cameras and Netgear routers carrying Mirai variants or newer Moobot/Gafgyt derivatives send SYN floods and HTTP floods with residential source IPs. No BGP blackhole community matches these sources because there is no bad ASN to match.
The controls that actually work operate at a layer below IP reputation: connection-level behavioral signals, subnet clustering, TCP/TLS fingerprinting, and high-performance per-source tracking using eBPF and XDP. These are implementable at the Linux host edge without application changes.
Threat Model
- Credential stuffing via residential proxy API: Attacker purchases residential proxy bandwidth, feeds target URL to OpenBullet with a wordlist. Each thread is assigned a fresh residential IP. Login endpoint receives requests from different IPs at 1–10 req/s per IP. Per-IP rate limiting never triggers. The aggregate rate across the server is the only signal — visible only if you measure it.
- Distributed web scraping with per-IP request capping: Scraper sends one request per IP, rotates across a pool of 50,000 IPs. From the server’s perspective, each IP has a clean request history. No rate-limit threshold is reached. The content is harvested at full fidelity across the residential pool.
- IoT botnet HTTP flood: A Mirai-derived botnet with 200,000 residential nodes (compromised home routers) sends HTTP GET floods to a target web server. Source IPs are distributed across residential ISP space globally. Volumetric — 400,000 requests per second peak. No single IP exceeds a reasonable per-IP limit. Application-layer WAF rules that fire on per-IP thresholds are useless.
- Residential proxy SYN state exhaustion: Residential exit nodes send SYN packets at volume with varying source addresses. The target’s conntrack table fills. Legitimate connections begin failing. No individual source IP is anomalous.
- Out of scope: Datacenter proxy traffic (easily blocked by ASN), Tor exit nodes (published list), known VPN ranges (covered by IP reputation). This threat model concerns only residential and IoT-origin traffic where the source IP itself is not a useful signal.
Hardening Configuration
1. Baseline nftables Table Structure
Before adding rate-limiting rules, establish a clean table structure that separates concerns and allows dynamic set management:
#!/usr/bin/env bash
# /etc/nftables-bot-mitigation.sh
# Load with: nft -f /etc/nftables-bot-mitigation.sh
# Flush existing bot mitigation table if present
nft delete table inet bot_mitigation 2>/dev/null || true
nft add table inet bot_mitigation
# Ingress hook — high priority, runs before the main filter table
nft add chain inet bot_mitigation ingress \
'{ type filter hook input priority -100; policy accept; }'
2. Per-/24 Subnet Rate Limiting
Residential proxy networks, despite their geographic distribution, cluster within /24 subnets because residential ISPs allocate blocks to geographic serving areas. A /24 subnet contains 254 usable addresses. If 10 distinct residential IPs from the same /24 contact your login endpoint within a minute, that is a reliable signal — legitimate users do not cluster that way.
nftables meter objects implement per-key rate tracking in kernel space with no userspace involvement:
# Per-/24 rate limit: more than 60 HTTP connections/minute from a subnet triggers
# This uses the nftables meter syntax — kernel maintains hash table of subnet → rate state
nft add rule inet bot_mitigation ingress \
tcp dport { 80, 443 } \
ip saddr and 255.255.255.0 \
meter subnet_http_rate \
'{ ip saddr and 255.255.255.0 limit rate over 60/minute burst 15 packets }' \
counter drop
# Tighter limit for login and API authentication endpoints
# Requires destination IP match if terminating TLS on the host itself
# For reverse proxy setups, apply this to the proxy listener port
nft add rule inet bot_mitigation ingress \
tcp dport 8443 \
ip saddr and 255.255.255.0 \
meter subnet_auth_rate \
'{ ip saddr and 255.255.255.0 limit rate over 20/minute burst 5 packets }' \
counter drop
The and 255.255.255.0 construct applies a bitwise AND to the source IP before using it as the meter key, collapsing the full /32 into the /24 network address. The kernel meter maintains one rate state per distinct /24 address, not per individual IP.
For IPv6 traffic, residential proxy operators increasingly support IPv6 exit nodes. Apply the same logic to /48 prefixes (residential ISPs typically assign /48 to individual subscribers):
nft add rule inet bot_mitigation ingress \
tcp dport { 80, 443 } \
ip6 saddr and ffff:ffff:ffff:: \
meter subnet_http_rate_v6 \
'{ ip6 saddr and ffff:ffff:ffff:: limit rate over 60/minute burst 15 packets }' \
counter drop
3. Dynamic Blocklist with Automatic Expiry
Static blocklists require manual management. nftables dynamic sets with timeout flags allow rules to add entries at line rate and have them auto-expire without any userspace daemon:
# Persistent sets that survive nft reload (stored in kernel, referenced by name)
nft add set inet bot_mitigation blocked_subnets_v4 \
'{ type ipv4_addr; flags dynamic, timeout; timeout 1h; }'
nft add set inet bot_mitigation blocked_subnets_v6 \
'{ type ipv6_addr; flags dynamic, timeout; timeout 1h; }'
# Check membership first (fast path for already-blocked subnets)
nft add rule inet bot_mitigation ingress \
ip saddr and 255.255.255.0 \
@blocked_subnets_v4 \
counter drop
# Escalated rate: add to blocklist when subnet exceeds 500 connections/minute
# Lower threshold fires first (drop); this fires on extreme abuse (auto-block for 30 min)
nft add rule inet bot_mitigation ingress \
tcp dport { 80, 443 } \
ip saddr and 255.255.255.0 \
meter aggressive_subnet_check \
'{ ip saddr and 255.255.255.0 limit rate over 500/minute }' \
add @blocked_subnets_v4 \
'{ ip saddr and 255.255.255.0 timeout 30m }' \
drop
The add @set { element timeout N } construct in nftables inserts the element if not present, or refreshes its timeout if already there. A subnet actively abusing will keep getting its 30-minute window refreshed at each hit until it stops. A subnet that backs off will have its entry expire and regain clean status automatically — no manual intervention needed.
4. Complete nftables Configuration File
The inline nft add rule commands above are useful for understanding but unwieldy in production. The equivalent /etc/nftables.d/bot-mitigation.nft file:
table inet bot_mitigation {
set blocked_subnets_v4 {
type ipv4_addr
flags dynamic, timeout
timeout 1h
# Allow up to 65536 entries; LRU eviction when full
size 65536
}
set blocked_subnets_v6 {
type ipv6_addr
flags dynamic, timeout
timeout 1h
size 16384
}
# Static allowlist: never rate-limit your own monitoring IPs or CDN ranges
set allowlist_v4 {
type ipv4_addr
flags interval
elements = {
# Add your CDN origin ranges, monitoring IPs here
# 203.0.113.0/24, # Example: monitoring
}
}
chain ingress {
type filter hook input priority -100
policy accept
# Allowlist passes immediately — no rate limiting
ip saddr @allowlist_v4 accept
# Block already-known bad subnets without spending meter state
ip saddr and 255.255.255.0 @blocked_subnets_v4 \
counter name "blocked_subnet_hits" drop
ip6 saddr and ffff:ffff:ffff:: @blocked_subnets_v6 \
counter name "blocked_subnet_v6_hits" drop
# Per-/24 rate limit for HTTP/HTTPS — 60 new connections/minute
tcp dport { 80, 443 } \
ip saddr and 255.255.255.0 \
meter subnet_http_rate size 32768 \
{ ip saddr and 255.255.255.0 limit rate over 60/minute burst 15 packets } \
counter drop
# Tighter limit for auth endpoints
tcp dport { 8443, 8080 } \
ip saddr and 255.255.255.0 \
meter subnet_auth_rate size 16384 \
{ ip saddr and 255.255.255.0 limit rate over 20/minute burst 5 packets } \
counter drop
# Aggressive escalation to blocklist
tcp dport { 80, 443 } \
ip saddr and 255.255.255.0 \
meter aggressive_check size 32768 \
{ ip saddr and 255.255.255.0 limit rate over 500/minute } \
add @blocked_subnets_v4 \
{ ip saddr and 255.255.255.0 timeout 30m } \
drop
# IPv6 equivalent
tcp dport { 80, 443 } \
ip6 saddr and ffff:ffff:ffff:: \
meter subnet_http_rate_v6 size 8192 \
{ ip6 saddr and ffff:ffff:ffff:: limit rate over 60/minute burst 15 packets } \
counter drop
}
}
Load and verify:
nft -c -f /etc/nftables.d/bot-mitigation.nft # dry-run check
nft -f /etc/nftables.d/bot-mitigation.nft
nft list table inet bot_mitigation
nft list set inet bot_mitigation blocked_subnets_v4 # inspect dynamic entries
5. TCP Stack Fingerprinting with p0f and nftables Marks
Residential proxy networks route HTTP/HTTPS connections through real residential hosts, but the TCP characteristics of those hosts often mismatch the User-Agent header the bot is sending. A bot claiming to be Chrome on Windows 11 but arriving with a Linux TCP fingerprint (different initial window size, different TCP option order) is a signal.
p0f performs passive OS fingerprinting by inspecting the TCP SYN packet characteristics: initial window size, TTL, MSS, window scaling factor, and TCP option order (SACK, timestamps, NOP ordering). It runs passively on the interface and exposes results via a Unix socket:
# Run p0f on the ingress interface, writing fingerprint results to socket
# -d daemonizes; -o logs to file for SIEM ingestion
p0f -i eth0 -s /var/run/p0f.sock -d \
-o /var/log/p0f/p0f.log
# Verify p0f is classifying connections
p0f-client /var/run/p0f.sock 203.0.113.45
# Result: os=Windows, os_ver=10, link_type=Ethernet, distance=3
The integration path between p0f and nftables uses conntrack marks. A small daemon reads p0f’s socket and calls conntrack -U to mark connections where the TCP fingerprint is anomalous — a Linux fingerprint but User-Agent claims Windows, or the TTL suggests a device that is geographically far away for a “residential” IP that should resolve nearby:
#!/usr/bin/env python3
# /usr/local/bin/p0f-nft-marker.py
# Reads p0f API, marks suspicious connections via conntrack
import socket
import struct
import subprocess
P0F_SOCKET = '/var/run/p0f.sock'
SUSPICIOUS_MARK = 0x2 # nftables ct mark value for suspicious fingerprint
def query_p0f(src_ip, src_port, dst_ip, dst_port):
"""Query p0f API socket for connection fingerprint."""
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
sock.connect(P0F_SOCKET)
# p0f v3 query format: magic(4) + type(1) + addr_type(1)
# + src_ip(4) + dst_ip(4) + src_port(2) + dst_port(2)
query = struct.pack('!IBBIHHI',
0x50304601, # p0f v3 magic
0, # query type: by IP
4, # IPv4
struct.unpack('!I', socket.inet_aton(src_ip))[0],
struct.unpack('!I', socket.inet_aton(dst_ip))[0],
src_port, dst_port)
sock.send(query)
response = sock.recv(232)
sock.close()
return response
def mark_suspicious_connection(src_ip, src_port, dst_ip, dst_port):
"""Use conntrack to mark a suspicious connection."""
subprocess.run([
'conntrack', '-U',
'--src', src_ip, '--sport', str(src_port),
'--dst', dst_ip, '--dport', str(dst_port),
'--mark', str(SUSPICIOUS_MARK)
], capture_output=True)
In nftables, drop or rate-limit connections with the suspicious mark:
chain ingress {
# Drop connections marked as OS-fingerprint-suspicious
# (marked by p0f-nft-marker.py via conntrack)
tcp dport { 80, 443 } ct mark 0x2 counter drop
}
This does not scale to high-volume bot attacks on its own — the conntrack lookup is per-connection — but it is highly effective for targeted enforcement against lower-volume residential proxy credential stuffing where the bot operator has not taken the care to match TCP fingerprints to claimed User-Agents.
6. XDP Rate Limiting via eBPF
For volumetric IoT botnet traffic where throughput matters — SYN floods, HTTP floods from 200,000 compromised residential nodes — XDP (eXpress Data Path) runs a BPF program at the NIC driver level, before kernel networking, at line rate:
// rate_limit_xdp.c
// Compile: clang -O2 -g -target bpf -c rate_limit_xdp.c -o rate_limit_xdp.o
// Load: ip link set dev eth0 xdp obj rate_limit_xdp.o sec xdp
#include <linux/bpf.h>
#include <linux/if_ether.h>
#include <linux/ip.h>
#include <linux/tcp.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_endian.h>
// Rate limit: packets per second per /24 subnet
#define RATE_LIMIT_PPS 100
// Sliding window rate state per /24
struct rate_state {
__u64 window_start_ns;
__u32 count;
__u32 _pad;
};
// LRU hash: 1M entries, kernel evicts LRU when full — prevents map exhaustion
// under large residential proxy pools (400K+ /24 subnets possible)
struct {
__uint(type, BPF_MAP_TYPE_LRU_HASH);
__uint(max_entries, 1 << 20);
__type(key, __u32); // /24 network prefix (source IP & 0xFFFFFF00)
__type(value, struct rate_state);
} subnet_rate_state SEC(".maps");
static __always_inline __u32 ip_to_subnet(__u32 ip_be) {
// Mask to /24: keep top 24 bits, zero lower 8
return ip_be & bpf_htonl(0xFFFFFF00);
}
SEC("xdp")
int xdp_rate_limit(struct xdp_md *ctx) {
void *data = (void *)(long)ctx->data;
void *data_end = (void *)(long)ctx->data_end;
// Parse Ethernet header
struct ethhdr *eth = data;
if ((void *)(eth + 1) > data_end)
return XDP_PASS;
if (bpf_ntohs(eth->h_proto) != ETH_P_IP)
return XDP_PASS;
// Parse IP header
struct iphdr *ip = (void *)(eth + 1);
if ((void *)(ip + 1) > data_end)
return XDP_PASS;
// Only rate-limit TCP (HTTP/HTTPS) — not ICMP, UDP, etc.
if (ip->protocol != IPPROTO_TCP)
return XDP_PASS;
// Parse TCP header to check destination port
struct tcphdr *tcp = (void *)ip + (ip->ihl * 4);
if ((void *)(tcp + 1) > data_end)
return XDP_PASS;
__u16 dport = bpf_ntohs(tcp->dest);
if (dport != 80 && dport != 443)
return XDP_PASS;
__u32 subnet = ip_to_subnet(ip->saddr);
__u64 now_ns = bpf_ktime_get_ns();
struct rate_state *rs = bpf_map_lookup_elem(&subnet_rate_state, &subnet);
if (!rs) {
struct rate_state new_rs = {
.window_start_ns = now_ns,
.count = 1,
};
bpf_map_update_elem(&subnet_rate_state, &subnet, &new_rs, BPF_ANY);
return XDP_PASS;
}
// If window is older than 1 second, reset
if ((now_ns - rs->window_start_ns) > 1000000000ULL) {
rs->window_start_ns = now_ns;
rs->count = 1;
return XDP_PASS;
}
rs->count++;
if (rs->count > RATE_LIMIT_PPS) {
return XDP_DROP;
}
return XDP_PASS;
}
char _license[] SEC("license") = "GPL";
Build and load:
# Build dependencies: clang, libbpf-dev, linux-headers
clang -O2 -g -target bpf \
-I/usr/include/$(uname -m)-linux-gnu \
-c rate_limit_xdp.c -o rate_limit_xdp.o
# Load in native mode (preferred — uses driver XDP hook if available)
# Falls back to generic mode if driver doesn't support native XDP
ip link set dev eth0 xdp obj rate_limit_xdp.o sec xdp
# Verify it loaded
ip link show eth0 | grep xdp
# Inspect map statistics during an active attack
bpftool map show
bpftool map dump id <map_id>
# Remove XDP program
ip link set dev eth0 xdp off
The LRU hash map is critical: without LRU eviction, a sufficiently large residential proxy pool (100M IPs = ~400K /24 subnets) would exhaust the map. With BPF_MAP_TYPE_LRU_HASH the kernel evicts the least recently seen subnets, keeping the map bounded. Under an actual IoT botnet flood, active subnets cycle slowly enough that the LRU works correctly — a subnet sending traffic will have its entry refreshed before eviction.
To identify the top abusing subnets from the XDP map during an active attack:
bpftool map dump id <subnet_rate_state_id> -j \
| python3 -c "
import sys, json
data = json.load(sys.stdin)
for entry in sorted(data, key=lambda x: x['value']['count'], reverse=True)[:20]:
ip_int = int(entry['key'], 16)
count = entry['value']['count']
print(f'{(ip_int>>24)&0xff}.{(ip_int>>16)&0xff}.{(ip_int>>8)&0xff}.0/24 pps={count}')
"
This gives you the most actively-dropping /24 subnets in real time, which you can then add to the nftables blocklist with longer timeouts for sustained suppression.
7. BGP RTBH for Volumetric IoT Botnet DDoS
When an IoT botnet pushes traffic past what XDP can absorb at the host — which typically means multi-Gbps or NIC-saturating traffic — the response must be upstream: Remote Triggered Black Hole (RTBH) routing via BGP.
RTBH announces the victim prefix with a blackhole community tag. The transit provider’s edge routers match the community and drop traffic destined for that prefix before it reaches your network. This is a last resort (it also drops legitimate traffic to the target IP) but prevents infrastructure impact:
# Signal to upstream router via BGP (requires prior agreement with provider)
# Most transit providers honour community 65535:666 (RFC 7999 blackhole community)
# Some providers use proprietary communities — check your transit contract
# FRR (vtysh) — inject a /32 blackhole route tagged for upstream dropping:
# conf t
# ip route 203.0.113.5/32 blackhole
# router bgp 65001
# address-family ipv4 unicast
# redistribute static route-map RTBH_TAG
# !
# route-map RTBH_TAG permit 10
# match ip address prefix-list BLACKHOLE_TARGETS
# set community 65535:666 additive
# set local-preference 200
# For host-level testing without BGP propagation (drops at this host only):
ip route add 203.0.113.5/32 blackhole
ip route show 203.0.113.5
# blackhole 203.0.113.5
# Remove when attack subsides:
ip route del 203.0.113.5/32 blackhole
For IoT botnet DDoS where threat intelligence feeds identify specific compromised-router subnet ranges, you can combine XDP-level dropping with targeted RTBH for the heaviest-contributing /24 ranges identified via the bpftool inspection above.
8. Application-Layer Behavioral Signals That Survive IP Rotation
Kernel-level controls buy time and reduce noise but do not solve the underlying problem: if every request comes from a clean IP, you need signals that are not IP-derived. The most reliable application-layer signals that persist across IP rotation are connection-level, not HTTP-level.
TLS session resumption rate: Legitimate users’ browsers and mobile apps resume TLS sessions aggressively to avoid full handshake overhead. Bot operators using residential proxies almost never support session resumption — each request from a new IP is a new connection with a full TLS handshake. Measure the $ssl_session_reused variable in nginx:
# /etc/nginx/nginx.conf
http {
log_format bot_analysis '$remote_addr '
'$ssl_session_reused ' # r = resumed, . = new handshake
'"$http_user_agent" '
'$request_time '
'$upstream_response_time '
'$request_uri';
access_log /var/log/nginx/bot_analysis.log bot_analysis;
# TLS session cache — sessions must be cached for the reuse signal to fire
ssl_session_cache shared:SSL:50m;
ssl_session_timeout 1d;
}
Parse $ssl_session_reused to identify suspicious IPs:
# Percentage of requests with new TLS sessions (not resumed) per source IP
# Normal browser traffic: 20-40% new sessions
# Residential proxy bot traffic: 95-100% new sessions (no session reuse across IPs)
awk '{print $1, $2}' /var/log/nginx/bot_analysis.log \
| awk '
$2 == "." { new[$1]++ }
$2 == "r" { reused[$1]++ }
END {
for (ip in new) {
total = new[ip] + reused[ip]+0
if (total > 10)
printf "%s new=%d reused=%d pct_new=%.1f\n",
ip, new[ip], reused[ip]+0, (new[ip]/total)*100
}
}
' | sort -k4 -rn | head -20
Feed high-confidence bot IPs (100% new TLS sessions, more than 10 requests) into the nftables blocklist:
# /usr/local/bin/feed-bot-blocklist.sh
# Run every 5 minutes via cron or systemd timer
awk '{print $1, $2}' /var/log/nginx/bot_analysis.log \
| awk '
$2 == "." { new[$1]++ }
$2 == "r" { reused[$1]++ }
END {
for (ip in new) {
total = new[ip] + reused[ip]+0
if (total >= 15 && (new[ip]/total)*100 > 95)
print ip
}
}
' \
| while IFS= read -r ip; do
# Add the /24 to the dynamic blocklist with 1-hour timeout
subnet=$(python3 -c "
import ipaddress, sys
net = ipaddress.IPv4Network(sys.argv[1] + '/24', strict=False)
print(str(net.network_address))
" "$ip")
nft add element inet bot_mitigation blocked_subnets_v4 \
"{ ${subnet} timeout 1h }" 2>/dev/null || true
done
Request timing distribution: Human users have Poisson-distributed inter-request intervals. Bots have either fixed intervals (timer-based) or burst-then-pause patterns from thread pool cycling. Collect $request_time distributions per User-Agent fingerprint and alert on unusually regular or burst-simultaneous request patterns. This signal is harder to extract at the kernel layer — it belongs in your SIEM or a stream processor consuming the nginx log — but it identifies bot traffic that has been configured to match TCP fingerprints and use genuine residential session handling.
Expected Behaviour After Hardening
After deploying the nftables per-/24 rate limiting: a credential stuffing campaign using 10,000 residential IPs across 200 /24 subnets generates 50 login attempts per /24. At the 20/minute limit for auth endpoints with a burst of 5, the first 20 attempts per /24 pass. The remaining 30 per /24 are dropped in kernel before they reach nginx. The attacker’s effective throughput drops to roughly 40% of intended rate. The realistic adversary response is to spread across more /24 subnets, which grows your meter table but does not bypass the per-/24 limit. The attack becomes more expensive; it does not disappear. That is the realistic outcome of kernel-level controls against residential proxy operators with large pools.
After loading the XDP program during a volumetric IoT botnet flood: SYN and HTTP packets from high-rate /24 subnets (exceeding 100 packets per second from a single /24) are dropped at the NIC before they enter the TCP stack. The conntrack table does not fill. Kernel CPU is not consumed by skb processing for flood traffic. Legitimate traffic passes normally because no legitimate /24 subnet legitimately exceeds 100 new TCP connections per second to a single destination.
After configuring nginx TLS session logging and the five-minute cron job feeding the blocklist: bot operator IPs — which never reuse TLS sessions — accumulate in blocked_subnets_v4 with 1-hour timeouts. Coverage is partial and delayed (each /24 enters the blocklist only after generating enough requests to trigger the log analysis), but the combination of kernel rate limiting and application-layer feedback creates a self-improving loop. Subnets that fall below the rate threshold and stop generating log entries have their blocklist entries expire automatically.
Trade-offs and Operational Considerations
Per-/24 collateral damage is unavoidable: A /24 contains 254 addresses. A residential proxy operator routes through real residential IPs. If your rate limit fires for a /24, legitimate users sharing that subnet will also be rate-limited until the window resets. In practice the probability is low during any given attack window, but not zero. Use burst values generously (burst 15 on a 60/minute limit means a short legitimate burst is absorbed), keep auto-block timeouts short (15–30 minutes), and never apply per-/24 limits to emergency services, financial institution ranges, or health system networks that you have identified as critical customers. The allowlist_v4 set exists for exactly this purpose.
XDP bypasses nftables: An XDP program loaded at the driver hook runs before netfilter/nftables processing. XDP_PASS forwards the packet into the normal stack; XDP_DROP discards it before nftables ever sees it. The two layers are not redundant — they are complementary. XDP handles the volumetric threshold (rate in packets per second); nftables handles the stateful blocklist (sustained abuse across multiple windows). If XDP is removed during an attack, nftables becomes the sole line of defense. Design for that case: ensure nftables limits are low enough to protect the host independently.
Meter state memory is finite: The size parameter on nftables meters caps kernel memory use. At the default of 32,768 entries and approximately 50 bytes per entry, this is 1.5 MB — negligible. When the meter is full, new /24 subnets are not tracked and therefore not rate-limited. Size your meters conservatively for expected active subnets, not maximum possible. 32,768 distinct /24 subnets simultaneously active is sufficient for all but the most extreme distributed attacks.
TLS session reuse feedback is asynchronous: The log-analysis blocklist feedback loop operates with a 5-minute lag at minimum. Kernel rate limiting is synchronous — meters fire instantly at line rate. Do not rely on the log-analysis loop for real-time attack response. It supplements the kernel controls by adding IPs that have demonstrated bot behavior to a longer-lived blocklist, but it is not a substitute for the nftables and XDP layers.
Failure Modes
Residential proxy operators rotate /24 subnets faster than blocklists update: BrightData and similar operators can refresh exit IPs every few minutes across a pool of 50M+ IPs spanning ~200,000 /24 subnets. If rotation is fast enough, the blocklist never accumulates meaningful coverage. In this scenario, per-/24 rate limiting is the primary control — the blocklist provides depth, not breadth. Accept this limitation explicitly: kernel-level controls reduce attack efficiency and increase operator cost, but do not provide complete exclusion against operators with very large, actively-rotating pools.
IPv6 /48 assumption is wrong for some ISPs: The premise that residential IPv6 subscribers receive /48 allocations holds for most North American and European ISPs, but some allocate /64 or /128 per subscriber. A /64 subnet rate limit applied to a single subscriber’s interface prefix is effectively a per-user limit, not a per-network limit — which is exactly the scenario you are trying to avoid with per-/24 logic. Before deploying IPv6 subnet rate limiting, inspect your access logs to determine the actual IPv6 prefix lengths your traffic uses and calibrate accordingly.
XDP program fails to load silently in CI/CD: The eBPF verifier rejects programs with unsafe memory accesses or unbounded loops. A verifier failure causes ip link set dev eth0 xdp obj ... to exit nonzero, leaving the host without XDP protection. If your deployment pipeline does not check the exit code and verify the XDP program loaded, an update that introduces a verifier failure removes XDP protection silently. Add a verification step:
ip link set dev eth0 xdp obj rate_limit_xdp.o sec xdp
ip link show eth0 | grep -q xdp || { echo "ERROR: XDP load failed"; exit 1; }
Add this to your deployment health checks and monitoring alerts.
Kernel-level controls are necessary but not sufficient: The most important operational failure mode is treating this configuration as a complete bot mitigation solution. A sophisticated residential proxy operator can spread attacks below the /24 rate threshold, configure genuine TLS session resumption, match TCP fingerprints to claimed User-Agents using OS-specific TCP stacks on the exit nodes, and vary request timing to mimic human behavior. Kernel-level defenses reduce the viable attack surface and increase operator cost; they do not provide a complete solution. For authentication specifically: the only control that actually defeats credential stuffing against residential proxy networks is multi-factor authentication that cannot be bypassed through IP rotation. Kernel controls buy time and reduce noise while MFA is deployed; they are not a substitute for it.