Network & API Security advanced 14 min read

Defending Against AI-Enhanced Adaptive DDoS Attacks

ddos ai traffic-analysis anomaly-detection ml scrubbing defence

Defending Against AI-Enhanced Adaptive DDoS Attacks

Problem

Traditional DDoS defences are built around fixed signatures and static thresholds. A scrubbing centre filters traffic matching known attack patterns — specific source ASNs, packet sizes, protocol flags, or request rates — and drops everything above a rate limit. This works well when attacks are predictable and persistent.

AI changes the attacker’s economics and capabilities. DDoS campaigns that previously required human operators to observe and adapt are now being automated with reinforcement learning models: the attack agent observes whether traffic is reaching the target (via probe traffic), adjusts its strategy when blocked, diversifies across vectors when a specific vector is mitigated, and maintains just enough legitimate-looking traffic to avoid threshold-based filtering while still overwhelming the target.

The documented evolution in 2024–2025:

Multi-vector adaptive campaigns. AI-driven botnets simultaneously probe multiple attack vectors (volumetric UDP flood, HTTP/2 rapid reset, QUIC amplification, slowloris) and allocate bot capacity toward whichever vector is most effective at the moment. When the defender mitigates one vector, the AI redistributes capacity to the next. Human defenders responding to static alerts cannot react fast enough.

Threshold-aware request flooding. AI attack agents calibrate request rates per source IP to stay below per-IP rate limits while collectively overwhelming the target. A rate limit of 100 requests/minute per IP with 10,000 bots produces 1 million requests per minute — each source is individually legitimate, but the aggregate is fatal.

Morphing application-layer attacks. HTTP-based attack traffic is increasingly indistinguishable from legitimate traffic at the packet level. AI-generated User-Agent strings, realistic request distributions, and randomised request paths defeat string-matching WAF rules. The attack traffic is statistically similar to real user traffic in headers, timing, and content patterns.

Feedback-loop C2 infrastructure. Attack C2 servers monitor target availability and adjust attack parameters in a closed loop. When a CDN edge node detects and rate-limits attack traffic, the C2 server observes the improved availability and increases attack intensity from unblocked sources.

The defensive implication is that static thresholds and signature-based filtering are necessary but no longer sufficient. Defences must also observe and adapt — using ML models that detect statistical anomalies in traffic rather than (or in addition to) matching patterns; dynamically adjusting rate limits based on current traffic patterns; and coordinating mitigation across multiple scrubbing layers simultaneously.

Target systems: internet-facing services with more than 10 Gbps peak legitimate traffic; SaaS platforms; financial services APIs; any service that has been targeted by volumetric DDoS in the past; services relying solely on static rate limits or rule-based WAFs for DDoS protection.

Threat Model

Adversary 1 — Adaptive volumetric campaign. A botnet of 50,000 devices runs an AI agent that monitors which attack vectors are reaching the target. It starts with UDP amplification, observes that the target’s scrubbing centre is filtering the amplification multipliers, switches to direct TCP SYN flood, observes rate limiting, switches to HTTPS flood with realistic browser fingerprints. Each switch happens within 30–60 seconds of the defender’s mitigation.

Adversary 2 — Threshold-calibrated request flood. 100,000 bots each send 95 requests/minute to a target with a 100 req/min per-IP rate limit. Each bot is individually below threshold; aggregate traffic is 9.5 million requests/minute, overwhelming the application tier. Rule-based rate limiting misses this attack entirely.

Adversary 3 — Slowloris variant with realistic pacing. HTTP connections are held open with AI-paced partial request sending that mimics slow mobile connections. The attack exhausts connection table space without triggering volumetric thresholds.

Adversary 4 — Application-layer with legitimate traffic camouflage. AI-generated requests include realistic User-Agent headers, proper TLS fingerprints, and request paths that match the target’s expected traffic distribution (derived from public analytics). Standard WAF rules that match attack signatures find nothing; the attack is indistinguishable from 10× normal traffic.

Configuration / Implementation

Step 1 — Establish a granular traffic baseline

ML-based anomaly detection requires a baseline. Capture multi-dimensional traffic metrics beyond simple volume:

# Deploy a traffic baselining tool at the network edge
# Using ntopng for multi-dimensional traffic profiling

# Key metrics to baseline per 5-minute window:
# - Total PPS and BPS by protocol
# - Unique source IPs per /24 prefix
# - TCP SYN:FIN:RST ratios
# - HTTP request distribution by path and method
# - TLS handshake rates vs. established connection rates
# - DNS query rates from each source
# - Geographic distribution of source IPs (for anomaly)

# With Prometheus + node_exporter + conntrack:
cat > /etc/prometheus/traffic-baseline.yml << 'EOF'
scrape_configs:
- job_name: 'conntrack_metrics'
  static_configs:
  - targets: ['localhost:9153']
  metric_relabel_configs:
  - source_labels: [__name__]
    regex: 'conntrack_.*'
    action: keep
EOF

Step 2 — Deploy ML-based anomaly detection alongside rule-based filtering

Use statistical anomaly detection that adapts to your baseline, not fixed thresholds:

# traffic_anomaly_detector.py
# Deployed as a sidecar to your scrubbing centre or as a standalone detection layer

import numpy as np
from sklearn.ensemble import IsolationForest
from dataclasses import dataclass
from typing import Optional
import time

@dataclass
class TrafficWindow:
    timestamp: float
    pps: float              # Packets per second
    bps: float              # Bits per second
    unique_src_ips: int     # Distinct source IPs in window
    syn_ratio: float        # SYN:total packet ratio
    new_conn_rate: float    # New connections per second
    req_per_src: float      # Mean requests per source IP
    geo_entropy: float      # Shannon entropy of source country distribution
    ua_entropy: float       # Shannon entropy of User-Agent distribution

class AdaptiveDDoSDetector:
    """
    Isolation Forest-based anomaly detector for DDoS detection.
    Adapts to traffic evolution over time using a rolling baseline.
    """
    
    def __init__(
        self,
        baseline_window: int = 2016,  # 1 week of 5-minute windows
        contamination: float = 0.01,  # Expected anomaly rate
        retrain_interval: int = 288,  # Retrain every 24h
    ):
        self.baseline_window = baseline_window
        self.contamination = contamination
        self.retrain_interval = retrain_interval
        self.baseline_data: list[list[float]] = []
        self.model = IsolationForest(
            contamination=contamination,
            random_state=42,
            n_estimators=200
        )
        self.windows_since_retrain = 0
        self.trained = False
    
    def _window_to_features(self, w: TrafficWindow) -> list[float]:
        return [
            w.pps, w.bps, w.unique_src_ips, w.syn_ratio,
            w.new_conn_rate, w.req_per_src, w.geo_entropy, w.ua_entropy
        ]
    
    def update(self, window: TrafficWindow) -> Optional[dict]:
        """Add a traffic window and return anomaly score if trained."""
        features = self._window_to_features(window)
        self.baseline_data.append(features)
        
        # Maintain rolling window
        if len(self.baseline_data) > self.baseline_window:
            self.baseline_data.pop(0)
        
        # Initial training after 1 day of data
        if len(self.baseline_data) == 288 and not self.trained:
            self._retrain()
        
        # Periodic retraining
        if self.trained:
            self.windows_since_retrain += 1
            if self.windows_since_retrain >= self.retrain_interval:
                self._retrain()
            
            # Score current window
            score = self.model.score_samples([features])[0]
            prediction = self.model.predict([features])[0]
            
            return {
                "timestamp": window.timestamp,
                "anomaly_score": float(score),
                "is_anomaly": prediction == -1,
                "severity": self._score_to_severity(score),
            }
        
        return None
    
    def _retrain(self):
        X = np.array(self.baseline_data)
        self.model.fit(X)
        self.trained = True
        self.windows_since_retrain = 0
    
    def _score_to_severity(self, score: float) -> str:
        if score < -0.2: return "CRITICAL"
        if score < -0.1: return "HIGH"
        if score < 0.0:  return "MEDIUM"
        return "NORMAL"

Step 3 — Integrate adaptive rate limiting

Replace static per-IP rate limits with dynamic ones that adjust to current traffic patterns:

# /etc/nginx/conf.d/adaptive-ratelimit.conf
# Nginx with dynamic rate limiting

# Base rate limit zones
limit_req_zone $binary_remote_addr zone=per_ip:20m rate=100r/m;
limit_req_zone $http_x_forwarded_for zone=per_real_ip:20m rate=100r/m;
limit_req_zone $server_name zone=per_server:10m rate=10000r/m;

# Connection limits
limit_conn_zone $binary_remote_addr zone=conn_per_ip:20m;

server {
    listen 443 ssl;
    
    # Apply adaptive rate limits (these are updated dynamically via API)
    limit_req zone=per_ip burst=20 nodelay;
    limit_req zone=per_server burst=500;
    limit_conn conn_per_ip 50;
    
    # Return 429 (not 503) for rate-limited requests
    limit_req_status 429;
    limit_conn_status 429;
    
    # Log rate limit hits for ML feedback
    log_format ratelimit '$remote_addr - $request - $status - $limit_req_status';
    access_log /var/log/nginx/ratelimit.log ratelimit if=$limit_req_status;
}

Python service to dynamically update Nginx rate limits based on ML detector output:

# adaptive_ratelimit_manager.py
import subprocess
import json

class AdaptiveRateLimitManager:
    """Dynamically adjusts Nginx rate limits based on attack detection."""
    
    BASE_LIMITS = {
        "per_ip_rate": "100r/m",
        "per_server_rate": "10000r/m",
        "conn_per_ip": 50,
    }
    
    ATTACK_LIMITS = {
        "MEDIUM": {
            "per_ip_rate": "30r/m",
            "per_server_rate": "5000r/m",
            "conn_per_ip": 20,
        },
        "HIGH": {
            "per_ip_rate": "10r/m",
            "per_server_rate": "2000r/m",
            "conn_per_ip": 10,
        },
        "CRITICAL": {
            "per_ip_rate": "5r/m",
            "per_server_rate": "500r/m",
            "conn_per_ip": 5,
        },
    }
    
    def apply_limits(self, severity: str) -> None:
        limits = self.ATTACK_LIMITS.get(severity, self.BASE_LIMITS)
        
        # Update Nginx config via include file and reload
        config = f"""
limit_req_zone $binary_remote_addr zone=per_ip:20m rate={limits['per_ip_rate']};
limit_req_zone $server_name zone=per_server:10m rate={limits['per_server_rate']};
limit_conn_zone $binary_remote_addr zone=conn_per_ip:20m;
"""
        with open("/etc/nginx/conf.d/dynamic-limits.conf", "w") as f:
            f.write(config)
        
        subprocess.run(["nginx", "-s", "reload"], check=True)
        print(f"Applied {severity} rate limits: {limits}")

Step 4 — Deploy at multiple scrubbing layers

AI-adaptive attacks probe individual scrubbing layers. Multi-layer defence reduces the feedback signal the attacker receives:

Layer 1: BGP anycast / upstream scrubbing centre
         → Volumetric filtering (Gbps-scale)
         → GeoIP-based blocking for attack-source regions
         → Protocol validation (malformed packets)

Layer 2: CDN edge (Cloudflare, AWS CloudFront)
         → Rate limiting per IP / ASN
         → Challenge pages for suspicious traffic
         → ML-based bot detection (JA4 fingerprinting)

Layer 3: Load balancer (nginx, HAProxy, Envoy)
         → Application-layer rate limiting (adaptive)
         → Connection table limits
         → Slow HTTP attack mitigation

Layer 4: Application tier
         → Per-user/session rate limiting
         → Circuit breaker for downstream services
         → Graceful degradation under load

The key: each layer applies independent mitigation. When the ML detector at Layer 3 sees an anomaly, it can signal Layer 1 to apply upstream filtering — reducing the feedback the attacker gets from their probe traffic.

Step 5 — Monitor for adaptive attack signatures

# Prometheus alerting rules for adaptive DDoS indicators

- alert: AdaptiveDDoSIndicator
  expr: |
    # Spike in unique source IPs with low request counts per IP (threshold-aware attack)
    (
      increase(nginx_connections_active[5m]) / on() group_left
      increase(nginx_http_requests_total[5m])
    ) > 2
    AND
    count by () (increase(nginx_http_requests_total{status="429"}[5m]) > 0) > 1000
  labels:
    severity: warning
  annotations:
    summary: "Possible threshold-calibrated DDoS — many sources near rate limit"

- alert: VectorShiftIndicator  
  expr: |
    # Rapid change in protocol distribution (attack switching vectors)
    abs(
      rate(node_network_receive_packets_total{device="eth0"}[5m]) -
      rate(node_network_receive_packets_total{device="eth0"}[5m] offset 5m)
    ) / rate(node_network_receive_packets_total{device="eth0"}[5m] offset 5m) > 0.5
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: "Traffic pattern shifted by >50% — possible attack vector change"

Expected Behaviour

Attack type Without ML detection With adaptive defence
Threshold-calibrated flood 100K bots at 95 req/min pass static limits Anomaly detected via req_per_src entropy; dynamic limits tightened
Multi-vector campaign First vector mitigated; attacker pivots freely ML detects traffic pattern shift; cross-layer mitigation triggered
Morphing HTTP attack Passes WAF signature rules UA entropy anomaly detected; challenge page deployed
Slowloris variant Fills connection table conn_per_ip limit tightened dynamically; incomplete connection timeout reduced

Trade-offs

Aspect Benefit Cost Mitigation
ML anomaly detection Catches attacks that evade fixed thresholds Initial false positive rate during baseline establishment Use 1-week baseline before enabling automated mitigation; start with alert-only mode
Dynamic rate limit reduction Reduces attack impact quickly May throttle legitimate traffic during attack ramp-up Use tiered response: MEDIUM limits reduce rate to 30%; CRITICAL to 5%; implement user-identifiable sessions to exempt authenticated users
Multi-layer scrubbing Makes probe-and-adapt harder for attacker Adds latency at each layer; complex to coordinate Test each layer independently; measure added latency; accept trade-off for high-value services
Adaptive retraining Model stays current with traffic evolution Attack traffic in training data can shift baseline (data poisoning) Exclude confirmed attack windows from retraining; use a separate baseline dataset

Failure Modes

Failure Symptom Detection Recovery
ML model trained on attack traffic Baseline shifts; attacks no longer flagged as anomalies Historical attack flags disappear from SIEM; attack events score as “NORMAL” Exclude attack-labelled windows from retraining dataset; maintain a static reference baseline that is never overwritten
Dynamic rate limiting triggers on flash crowd Legitimate traffic spike (viral content, breaking news) throttled; user impact 429 rate spike + normal UA/geo distribution; support tickets Implement exemption for authenticated sessions; use signed cookies to distinguish known users from bots
Single scrubbing layer feedback exploited Attacker observes clean probe traffic, increases attack precisely to just below detection threshold Attack visible in logs at steady sub-threshold rate Remove single-layer probe feedback; apply upstream scrubbing even for sub-threshold traffic at certain source ASNs
Anomaly detector retraining delay New daily traffic pattern (overnight batch jobs) initially flagged as DDoS Alert fires on expected batch job; high false positive rate Account for time-of-day patterns in feature engineering; add hour_of_day as a feature to the baseline