Hardening Linux TCP/IP Stacks Against Passive OS Fingerprinting

Problem

Passive OS fingerprinting identifies the operating system and kernel version of a host by analysing the characteristics of its network traffic — without sending a single active probe. Tools like p0f, Zeek’s fingerprinting module, Satori, and the passive fingerprinting built into nmap’s OS detection engine can identify the OS to within a kernel minor version by examining:

• Initial TTL values — Linux defaults to 64, Windows to 128, Cisco IOS to 255
• TCP window size — default window sizes differ by OS and version (Linux 5.x uses 64240, earlier versions used 29200 or 65535)
• TCP options and their ordering — the order of MSS, SACK, timestamps, NOP, and window scale options in a SYN packet is a reliable OS fingerprint
• IP ID randomisation quality — Linux uses per-connection random IDs; Windows uses incrementing counters; some embedded systems use constant 0
• TCP timestamp behaviour — whether timestamps are present, their frequency, and whether they increment linearly or randomly
• TCP SACK behaviour — whether SACK is advertised and whether SACK blocks are generated correctly

Passive fingerprinting is significant as a pre-exploitation reconnaissance technique. An attacker who can passively observe network traffic to a target — from a compromised adjacent system, from a cloud provider’s network, or via a BGP route through a monitored AS — can enumerate the kernel version of every host without triggering IDS alerts. This directs exploit selection: a host identified as Linux 5.15 on Ubuntu 22.04 narrows the CVE list significantly.

The complementary concern is that application-layer tools increasingly perform passive fingerprinting of their clients. Load balancers, WAFs, CDNs, and fraud detection systems fingerprint the TCP stack of connecting clients to detect bots, proxies, and VPN users — the same signals that JA3/JA4 analyse at the TLS layer, but at the TCP level. Hardening these signals improves privacy and frustrates detection evasion profiling.

Linux provides several sysctl settings and network configuration options that directly affect the fingerprint it presents. None completely defeat a determined fingerprinter — the OS can still often be identified to a family — but they significantly reduce the precision and increase the noise, forcing attackers to use active probing (which is detectable) to confirm their hypothesis.

Target systems: Linux 4.19–6.12 on internet-facing servers, Kubernetes ingress nodes, API gateways, bastion hosts, and any service where reconnaissance quality is a security concern; equally applicable to privacy-sensitive applications where client fingerprinting is a concern.

Threat Model

Adversary 1 — Network-positioned passive observer. Access level: ability to passively observe traffic to the target (BGP route analysis, compromised adjacent host, cloud provider network monitoring). Objective: identify OS and kernel version of target servers without active scanning; select appropriate exploits; time attacks for when patching is known to lag.

Adversary 2 — CDN/WAF fingerprint correlation. Access level: operates a CDN or security service that the target’s traffic transits. Objective: fingerprint the target’s kernel version to correlate with known vulnerability windows and sell this intelligence to threat actors, or to improve bot detection accuracy.

Adversary 3 — LAN-based attacker. Access level: access to the same network segment as the target (ARP-level visibility). Objective: passively fingerprint all hosts on the segment to build an OS inventory before launching a targeted attack.

Adversary 4 — Log analysis post-compromise. Access level: attacker who has already compromised one host. Objective: use p0f or similar tool to passively fingerprint all other hosts that have communicated with the compromised host via its connection logs — mapping the internal network topology without active scanning.

Without hardening: OS and kernel version are identifiable from a single SYN packet with high confidence. With hardening: fingerprint precision is degraded; attacker must fall back to active probing, which is detectable and slower.

Configuration / Implementation

Step 1 — Verify current fingerprint baseline

# Install p0f to test your own fingerprint
apt install p0f  # Debian/Ubuntu
# or: tcpdump + offline p0f database

# Capture your own SYN packets and run through p0f
tcpdump -i eth0 -w /tmp/capture.pcap 'tcp[tcpflags] & tcp-syn != 0 and tcp[tcpflags] & tcp-ack == 0' -c 20
p0f -r /tmp/capture.pcap 2>/dev/null | head -20

# Alternatively, use nmap's OS detection passively via a scan of yourself from another host
nmap -O --osscan-limit your.server.ip 2>/dev/null | grep "OS details\|Running:"

# Check current TCP SYN options
python3 - <<'EOF'
import socket, struct

# Craft a raw SYN and examine the options
# Or just use ss to check defaults
import subprocess
result = subprocess.run(['ss', '-tin', 'dport', '= :443'], capture_output=True, text=True)
print(result.stdout[:2000])
EOF

Step 2 — Randomise IP ID and TTL

The IP ID field and initial TTL are the most reliable passive fingerprint signals:

# /etc/sysctl.d/90-fingerprint-hardening.conf

# Randomise IP TTL slightly — makes TTL-based OS detection less reliable
# Linux default is 64; this adds random variation in range [64-127]
# Not a standard sysctl — achieved via iptables MARK + routing instead (see Step 4)

# Disable IP timestamps in ICMP (reduces fingerprint surface from ICMP echo)
net.ipv4.icmp_echo_ignore_all = 0         # Keep responding (disabling is conspicuous)
# ICMP rate limiting reduces timing-based fingerprint quality
net.ipv4.icmp_ratelimit = 1000
net.ipv4.icmp_ratemask = 88089

# Randomise IP ID sequences (Linux already does this per-connection; verify)
# Linux 3.18+ uses random IDs by default — confirm:
# grep "ipid" /proc/sys/net/ipv4/ 2>/dev/null

Step 3 — Normalise TCP window size and options

The TCP initial window size is a strong fingerprint. Normalise it toward a common value:

# /etc/sysctl.d/90-fingerprint-hardening.conf

# TCP window scaling — keep enabled but normalise the initial size
net.ipv4.tcp_window_scaling = 1

# Set initial receive window size — use a common value that appears in many OSes
# Default Linux 5.x is 64240; Windows uses 65535; many CDNs use 65535
# Setting to 65535 makes Linux look more generic
net.core.rmem_default = 212992
net.core.rmem_max = 212992
net.ipv4.tcp_rmem = 4096 87380 6291456

# TCP timestamps — disabling reduces fingerprint surface but harms PAWS and RTT measurement
# Consider the trade-off: leave enabled on high-speed links, disable if fingerprint resistance is priority
net.ipv4.tcp_timestamps = 0  # Disabling removes a reliable fingerprint signal

# TCP SACK — keep enabled (disabling degrades performance significantly)
net.ipv4.tcp_sack = 1

Apply:

sysctl --system
# Verify timestamp setting
sysctl net.ipv4.tcp_timestamps
# Should show: net.ipv4.tcp_timestamps = 0

Step 4 — Normalise TTL with iptables PREROUTING

Use iptables HMARK or NFQUEUE to introduce TTL variation that frustrates TTL-based fingerprinting:

# Normalise outbound TTL to 128 (mimics Windows) or 64 (standard Linux)
# Using iptables TTLINC/TTLDEC targets to normalise all outbound TTL to a fixed value

# Option 1: Fix TTL to 128 (blends with Windows traffic in mixed environments)
iptables -t mangle -A POSTROUTING -j TTL --ttl-set 128

# Save
iptables-save > /etc/iptables/rules.v4

# For IPv6
ip6tables -t mangle -A POSTROUTING -j HL --hl-set 128
ip6tables-save > /etc/iptables/rules.v6

# Option 2: Randomise TTL slightly (64 ± N) using HMARK to add entropy
# More complex; use if you want to specifically resist TTL normalisation detection

Step 5 — Deploy pf/nftables fingerprint normalisation at the network edge

For ingress infrastructure, use nftables to normalise TCP flags and options that appear in incoming connections (preventing your servers’ fingerprints from being inferred from response patterns):

# /etc/nftables-fingerprint.conf
# Normalise outbound TCP to reduce OS fingerprint precision

table inet fingerprint_normalize {
  chain postrouting {
    type filter hook postrouting priority 100; policy accept;

    # Normalise TCP MSS to a standard value (masks actual kernel default)
    tcp flags syn tcp option maxseg size set 1460

    # Strip TCP timestamp option from outbound SYN-ACK
    # (removes one of the most reliable fingerprint signals)
    # Note: only practical if tcp_timestamps=0 is also set
  }

  chain output {
    type filter hook output priority 0; policy accept;

    # Drop any ICMP type 13 (timestamp request) — timing fingerprint prevention
    icmp type timestamp-request drop
    icmp type timestamp-reply drop
  }
}

nft -f /etc/nftables-fingerprint.conf
# Persist
echo "include \"/etc/nftables-fingerprint.conf\";" >> /etc/nftables.conf

Step 6 — Harden TCP SYN options ordering (kernel compile option)

The ordering of TCP options in a SYN packet is the most reliable fingerprint signal and is determined at kernel compile time. For organisations building custom kernels:

# Check current TCP options ordering in your kernel
# It will show in a SYN capture: MSS, SACK, Timestamps, NOP, WS (standard Linux)

tcpdump -i lo -w - 2>/dev/null & \
  curl -s http://localhost/ > /dev/null 2>&1
  
tcpdump -r - -A 2>/dev/null | grep -A5 "TCP"

# Options reordering requires kernel patch or a TCP proxy layer.
# For most deployments, disabling timestamps (Step 3) and normalising
# window size (Step 3) provides adequate fingerprint degradation without
# requiring a kernel rebuild.

For Kubernetes deployments, place fingerprint normalisation at the ingress:

# Nginx ingress with TCP normalisation snippet
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
data:
  # Configure nginx to manipulate TCP options in responses
  # This affects the fingerprint seen by external observers
  proxy-connect-timeout: "10"
  # Use keep-alive settings that don't leak kernel version timing
  keep-alive: "75"
  keep-alive-requests: "100"

Step 7 — Verify fingerprint improvement

# Test from an external vantage point after applying controls
# Using nmap from a different host:
nmap -O --osscan-limit YOUR_SERVER_IP 2>/dev/null

# Before hardening: "OS details: Linux 5.15 (Ubuntu 22.04)"
# After hardening: "OS details: Linux 4.15 - 5.19" (version range widened)
# Or: "OS details: No exact OS matches" (best case)

# Using p0f passive capture (run on your server, observe your own traffic):
# After disabling timestamps and normalising window:
# p0f should show either incorrect identification or a wider confidence range

# Verify TTL normalisation
ping -c 3 YOUR_SERVER_IP | grep ttl
# Should show ttl=128 (if you set TTL to 128 in iptables)

Expected Behaviour

Trade-offs

Failure Modes

Related Articles

• TLS Fingerprinting JA3/JA4 — fingerprinting at the TLS layer, complementary to TCP-layer passive fingerprinting
• JA4 Fingerprint Evasion and Bot Detection — the attacker’s perspective on fingerprint evasion relevant to understanding the defensive signal
• Linux TCP/IP Stack Hardening — broader TCP/IP hardening that overlaps with fingerprint reduction
• HTTP Security Headers — complementary application-layer techniques for reducing information disclosure
• Encrypted Client Hello — hiding TLS fingerprints, the TLS equivalent of what this article does for TCP