Linux Kernel Sysctl Hardening: Securing TCP/IP Network Stack against SYN Floods

1. TCP/IP Network Stack Vulnerabilities & Kernel Defaults

Default Linux kernel parameters prioritize maximum compatibility across diverse network topologies over security. Out of the box, Linux kernels accept source-routed packets, process ICMP redirect messages, and allow unprivileged access to dmesg logs.

Attacker reconnaissance and denial-of-service vectors exploit default sysctl configurations to perform IP spoofing, TCP SYN floods, and man-in-the-middle packet redirection.

Hardening sysctl parameters via modular configuration files in /etc/sysctl.d/ ensures kernel security settings persist across reboots.

Tuning these low-level parameters provides defense-in-depth protection before packets even reach user-space applications or web servers.

# Inspect active kernel TCP syncookies status
sysctl net.ipv4.tcp_syncookies

2. Mitigating SYN Floods with TCP Syncookies

A TCP SYN flood attack sends thousands of TCP SYN packets with spoofed source IP addresses. The server allocates socket memory and waits for ACK responses that never arrive, quickly filling the SYN backlog queue.

Enabling net.ipv4.tcp_syncookies changes kernel behavior: when the SYN backlog queue fills, the kernel stops allocating memory state and instead encodes connection parameters into the TCP Sequence Number (cookie).

When the client sends the final ACK, the kernel verifies the sequence number cookie and opens the connection seamlessly.

Combining tcp_syncookies with increased tcp_max_syn_backlog buffers ensures high availability during distributed denial of service events.

# Enable SYN flood defenses in sysctl
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2

3. Preventing IP Spoofing via Reverse Path Filtering

IP spoofing attacks forge packet source IP addresses to impersonate trusted internal nodes or bypass firewall rules.

Reverse Path Filtering (rp_filter) checks whether incoming packets arrive on the same network interface that the kernel routing table would use to send a response back to that source IP address. If the packet arrives on a different interface, the kernel drops it immediately.

Enforcing strict reverse path filtering (rp_filter = 1) eliminates IP spoofing vectors across all interfaces.

This setting prevents attackers on adjacent network segments from injecting unauthorized traffic into established TCP sessions.

# Enable strict Reverse Path Filtering
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

4. Disabling ICMP Redirects & Source Routing

ICMP redirect messages allow routers to notify hosts of better routes. Malicious actors use forged ICMP redirects to alter host routing tables and perform man-in-the-middle (MitM) interception.

Similarly, Source Routing allows senders to specify the exact path a packet takes through a network, bypassing firewalls.

Production servers must explicitly disable ICMP redirect processing and source routing across all interfaces.

Disabling ICMP redirects prevents adversary-controlled gateways from hijacking traffic routed between cloud instances.

# Disable ICMP redirects and IP source routing
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0

5. Memory Protection, ASLR, & Symlink Hardening

In addition to network stack parameters, sysctl governs kernel memory protection and file system safety directives.

Enforcing Address Space Layout Randomization (kernel.randomize_va_space = 2) randomizes memory positions of stack, heap, and library mappings, making buffer overflow exploits unpredictable.

Restricting unprivileged dmesg access (kernel.dmesg_restrict = 1) prevents unprivileged users from reading kernel memory addresses.

Protected hardlinks and symlinks restrict unprivileged users from creating malicious symlink targets inside world-writable directories like /tmp.

# Memory protection and system safety
kernel.randomize_va_space = 2
kernel.dmesg_restrict = 1
kernel.kptr_restrict = 2
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
fs.protected_fifos = 2
fs.protected_regular = 2

6. Persistence & Live Reloading via /etc/sysctl.d/

Save all hardened parameters inside /etc/sysctl.d/99-security.conf to ensure configurations persist across system reboots.

Apply configuration updates dynamically in memory without restarting system services.

Verify active kernel parameter values using the sysctl command line utility.

# Apply all sysctl settings immediately
sysctl --system

# Verify active setting for ASLR
sysctl kernel.randomize_va_space

Written by Zyekh Abdul Qadir Jailani

Digital Forensics & Incident Response (DFIR) Specialist & Security Researcher specializing in Linux kernel hardening, threat hunting, and system security research.

LinkedIn • GitHub • Discord • Email • PGP Key