SYN Flood DDOS Defense in Kernel Code

Environment

Kernel 2.6.32

Knowledge

SYN Flood DDOS is still a big headache for system administrators, and we don't really have a solution for this kind of attack so far in the real world.
To start a connection between a server and a client, they have to perform "TCP 3-Way Handshake" first without any failure:

1. client -> server: SYN
2. server -> client: SYN-ACK
3. client -> server: ACK

* SYN/ACK are signal bits in TCP packages.
However, not everyone in the internet is friendly and follow every rule we have. An attacker may only send SYN-package (first step in the 3-Way Handshake) but with pretty large amount, the server then may run out of resource since there are too many half-open connections. And, the attacker can even apply random IP source in the packages so that the server is not able to distinguish the bad/good clients.

Preface

In Kernel code, there's a hash table with request socket queues as entries. Whenever the new sock request (SYN-package) comes, it's inserted into this queue-entry hash table.
However, of course, the capability is limit, Kernel has to dropped some request sockets to allow new sockets with the table is full. In the original Kernel code, it kills old sockets every 0.2 second in net/ipv4/inet_connection_sock.c:inet_csk_reqsk_queue_prune.
Comparing to the original code, in this last OSDI lab, we are not relied on the regular routine to drop old request sockets, but we are randomly dropping a socket in the table when a new socket request comes and the table is full.

Implementation

Modify the Original Pruning Routine

In net/ipv4/inet_connection_sock.c:inet_csk_reqsk_queue_prune add following in line 513:

        thresh = max_retries;
    

Randomly Dropping Socket when Is Full

After tracing the code, we know that the function reqsk_queue_is_full is called whenever the TCP program wants to add a new sock in the request queue. So we simply add some code in the this function.

static inline int reqsk_queue_is_full(struct request_sock_queue *queue)
{
    struct listen_sock *lopt = queue->listen_opt;
    struct request_sock **reqp, *req;
    int i, range, random;
    unsigned long now = jiffies;

    int r = (queue->listen_opt->qlen >> queue->listen_opt->max_qlen_log);

    printk("[OSDI] qlen = %d\n", lopt->qlen);
    if( r ) {

        range = lopt->nr_table_entries;
        printk("range = %d\n", range);

        random %= range;
        get_random_bytes(&random, sizeof(range));

        i = lopt->clock_hand;
        do {
            i = (i + 1) & (lopt->nr_table_entries - 1);
        } while (--random > 0);

        printk("i=%d first is killed\n", i);

        reqp=&lopt->syn_table[i];
        if ((req = *reqp) != NULL) {
            if (time_after_eq(now, req->expires)) {
                /* OSDI lab12 */
                reqsk_queue_unlink(queue, req, reqp);
                reqsk_queue_removed(queue, req);
                reqsk_free(req);
            }
        }
        lopt->clock_hand = i;

    }

    // get the result again
    return queue->listen_opt->qlen >> queue->listen_opt->max_qlen_log;
}
    

We first calculate the original result for is_full or not. If it's full, then we random pick on queue entry from the table and drop the first element of the queue.

Test

Method 1 - Use iptables (Firewall)

Use both simple user socket programs on two machines. One is the client, another is the server. Make sure they can connect at the beginning (the routing is okay between these two machines/VMs). Then, drop any packets from the server on the client by setting up the firewall so that we will only send SYN-packages to the server.

sudo iptables -A INPUT -s <SERVER_IP> -j REJECT

Method 2 - Use hping3

sudo hping3 -i u1 -S -c 10 <SERVER_IP>

Another DDOS Experiment in our Workshop

The attacked server (right-hand side) is having a high CPU usage while another server (left-hand side) is send SYN Flood packages.