Linux Asynchronous I/O Internals: io_uring vs. epoll in High-Throughput Network Engines

For over two decades, the backbone of high-performance Linux networking has been epoll. Introduced in Linux 2.5.44, epoll powered the C10K revolution, serving as the foundational event notification engine behind Nginx, Node.js (libuv), Redis, Envoy, Netty, and Go's runtime network poller.

However, modern hardware realities have dramatically shifted. With 100GbE/400GbE network interfaces, NVMe SSDs capable of millions of IOPS with microsecond latencies, and CPU architectures vulnerable to Meltdown/Spectre mitigation overheads, the classic readiness-based epoll model has hit a fundamental scalability ceiling.

Enter io_uring, created by Jens Axboe and merged into Linux kernel 5.1+. Rather than notifying user-space when a socket or file descriptor is ready to perform I/O, io_uring is a true asynchronous completion-based I/O framework. By sharing lockless, circular ring buffers between user space and kernel space via memory mapping (mmap), io_uring eliminates system call context switches, amortizes memory copies, and enables zero-syscall I/O.

In this systems engineering guide, we dissect the internal mechanics of epoll vs. io_uring, explore the anatomy of Submission and Completion Queues (SQ/CQ), analyze kernel-polling mode (IORING_SETUP_SQPOLL), and demonstrate how to build high-concurrency network servers in C and Rust.

The Readiness vs. Completion Paradigm

To understand why io_uring outperforms epoll, we must examine the difference between Readiness-based I/O and Completion-based I/O:

The Syscall Tax & The epoll Scalability Wall

In a high-throughput epoll event loop servicing 100,000 requests per second, the application must execute a cascade of system calls:

The epoll Execution Cascade (Repeated Context Switches):

User Space (Ring 3)                         Kernel Space (Ring 0)
───────────────────                         ────────────────────
1. epoll_wait() ────────[ Context Switch ]──────> Blocks until event
2. Returns ready FD <───[ Context Switch ]──────< Unblocks with events
3. read(fd, buf) ───────[ Context Switch ]──────> Copies data from socket buffer to buf
4. Returns bytes <──────[ Context Switch ]──────< Copy finished
5. [Process Data in User Space]
6. write(fd, out) ──────[ Context Switch ]──────> Copies out to socket send buffer
7. Returns count <──────[ Context Switch ]──────< Copy finished

Every single transition between user space and kernel space requires saving CPU registers, flushing Translation Lookaside Buffers (TLB) or executing KPTI (Kernel Page Table Isolation) trampolines, and executing branch predictors. When handling millions of IOPS on NVMe drives or 100GbE networks, up to 60% of CPU time is spent solely on syscall boundary transitions rather than business logic.

Furthermore, epoll fundamentally fails for standard disk files. Under Linux, regular disk file descriptors always report as "readable" and "writable" to epoll, even when the data is not in the page cache, causing synchronous storage stall bubbles that freeze the entire event loop.

io_uring Architecture: Lockless Dual-Ring Buffers

The core innovation of io_uring is the separation of I/O requests and I/O results into two lockless circular ring buffers mapped into memory shared between user space and the Linux kernel:

io_uring Memory-Mapped Architecture:

┌─────────────────────────────────────────────────────────────┐
│                        USER SPACE                           │
│                                                             │
│   1. Write Submission Queue Entry (SQE)                     │
│      [ Op: IORING_OP_READV | FD: 4 | Buffer: 0x7fff... ]    │
│      Update SQ Tail (Atomic store)                          │
│                                                             │
│   4. Read Completion Queue Entry (CQE)                      │
│      [ User_data: 0x42 | Res: 1024 bytes read | Flags: 0 ]  │
│      Update CQ Head (Atomic store)                          │
└──────────────┬───────────────────────────────▲──────────────┘
               │                               │
        SHARED MEMORY MAP (mmap - Zero Copy Boundary)
               │                               │
┌──────────────▼───────────────────────────────┴──────────────┐
│                       KERNEL SPACE                          │
│                                                             │
│   2. Kernel reads SQ Head -> Submits to NVMe / NIC Driver   │
│   3. Hardware completes I/O -> Kernel writes CQE to CQ Tail │
└─────────────────────────────────────────────────────────────┘

1. The Submission Queue (SQ)

The application acts as the Producer and the Linux kernel acts as the Consumer:

• The application fills one or more struct io_uring_sqe (Submission Queue Entries) detailing the desired operation (e.g., IORING_OP_ACCEPT, IORING_OP_READ, IORING_OP_SEND, IORING_OP_SPLICE).
• The application updates the ring buffer tail pointer using a release-memory barrier.
• Multiple requests (e.g. 64 independent socket reads and writes) are submitted in a single batch!

2. The Completion Queue (CQ)

The Linux kernel acts as the Producer and the application acts as the Consumer:

• When the hardware or kernel driver finishes an I/O operation, the kernel appends a struct io_uring_cqe (Completion Queue Entry) containing the return code (number of bytes read/written or negative error code) and the user-provided user_data identifier.
• The application inspects the CQ head, processes completions, and advances the head pointer without making a single system call.

Zero-Syscall I/O: Kernel Polling Mode (IORING_SETUP_SQPOLL)

In standard mode, after appending SQEs to the Submission Queue, the application calls io_uring_enter() to notify the kernel that new work is pending.

However, for ultra-low latency scenarios (e.g. high-frequency trading, real-time gaming, high-throughput caching), io_uring provides SQPOLL Mode:

SQPOLL Mode (True Zero-Syscall Operation):

User Space Thread:
While (true) {
    Append SQE directly to SQ ring buffer in RAM
    Atomic write: sq_tail++
    // ZERO SYSTEM CALLS!
}

Dedicated Kernel Thread (io_uring-sq):
While (true) {
    Polls SQ ring buffer in RAM for new SQEs
    Dispatches immediately to hardware driver
    Writes results to CQ ring buffer
    // ZERO CONTEXT SWITCHES!
}

Advanced Features: Fixed Buffers & Registered File Descriptors

io_uring goes far beyond basic async operations, offering hardware-accelerated zero-copy primitives:

1. Registered Buffers (IORING_REGISTER_BUFFERS)

In standard I/O, the kernel must map and pin the user buffer's virtual memory pages into physical RAM on every single request, then unpin them when complete. With io_uring_register_buffers(), the application pre-pins a pool of buffers once at startup. Subsequent reads and writes execute with zero page mapping overhead.

2. Registered Files (IORING_REGISTER_FILES)

Every standard file descriptor access requires the kernel to increment and decrement the file table reference count (causing atomic cache-line bouncing across CPU cores). By pre-registering an array of file descriptors, io_uring accesses file objects via direct array indexing, bypassing the file table lock entirely.

Hands-On: Building a High-Throughput Echo Server in C with liburing

Below is a production-grade implementation of a non-blocking TCP Echo Server built using liburing (the official C wrapper for io_uring):

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <netinet/in.h>
#include <liburing.h>

#define QUEUE_DEPTH 256
#define BUFFER_SIZE 2048
#define PORT 8080

enum EventType {
    EVENT_ACCEPT,
    EVENT_READ,
    EVENT_WRITE
};

struct ConnState {
    int fd;
    enum EventType type;
    char buffer[BUFFER_SIZE];
    struct iovec iov;
};

void add_accept_sqe(struct io_uring *ring, int server_fd, struct sockaddr_in *client_addr, socklen_t *client_len) {
    struct io_uring_sqe *sqe = io_uring_get_sqe(ring);
    io_uring_prep_accept(sqe, server_fd, (struct sockaddr *)client_addr, client_len, 0);
    
    struct ConnState *state = malloc(sizeof(struct ConnState));
    state->fd = server_fd;
    state->type = EVENT_ACCEPT;
    io_uring_sqe_set_data(sqe, state);
}

void add_read_sqe(struct io_uring *ring, int client_fd) {
    struct io_uring_sqe *sqe = io_uring_get_sqe(ring);
    struct ConnState *state = malloc(sizeof(struct ConnState));
    state->fd = client_fd;
    state->type = EVENT_READ;
    state->iov.iov_base = state->buffer;
    state->iov.iov_len = BUFFER_SIZE;

    io_uring_prep_readv(sqe, client_fd, &state->iov, 1, 0);
    io_uring_sqe_set_data(sqe, state);
}

void add_write_sqe(struct io_uring *ring, struct ConnState *state, int bytes_read) {
    struct io_uring_sqe *sqe = io_uring_get_sqe(ring);
    state->type = EVENT_WRITE;
    state->iov.iov_base = state->buffer;
    state->iov.iov_len = bytes_read;

    io_uring_prep_writev(sqe, state->fd, &state->iov, 1, 0);
    io_uring_sqe_set_data(sqe, state);
}

int main() {
    struct io_uring ring;
    io_uring_queue_init(QUEUE_DEPTH, &ring, 0);

    int server_fd = socket(AF_INET, SOCK_STREAM, 0);
    int opt = 1;
    setsockopt(server_fd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));

    struct sockaddr_in server_addr = {
        .sin_family = AF_INET,
        .sin_port = htons(PORT),
        .sin_addr.s_addr = INADDR_ANY
    };

    bind(server_fd, (struct sockaddr *)&server_addr, sizeof(server_addr));
    listen(server_fd, SOMAXCONN);
    printf("io_uring Echo Server listening on port %d\n", PORT);

    struct sockaddr_in client_addr;
    socklen_t client_len = sizeof(client_addr);
    add_accept_sqe(&ring, server_fd, &client_addr, &client_len);
    io_uring_submit(&ring);

    while (1) {
        struct io_uring_cqe *cqe;
        // Wait for at least 1 completion entry
        int ret = io_uring_wait_cqe(&ring, &cqe);
        if (ret < 0) {
            perror("io_uring_wait_cqe");
            break;
        }

        struct ConnState *state = (struct ConnState *)io_uring_cqe_get_data(cqe);
        int res = cqe->res;

        if (state->type == EVENT_ACCEPT) {
            int client_fd = res;
            if (client_fd >= 0) {
                // Register read for new client
                add_read_sqe(&ring, client_fd);
            }
            // Re-arm accept for subsequent connections
            add_accept_sqe(&ring, server_fd, &client_addr, &client_len);
            free(state);
        } else if (state->type == EVENT_READ) {
            if (res > 0) {
                // Echo data back to client
                add_write_sqe(&ring, state, res);
            } else {
                // Client disconnected or error
                close(state->fd);
                free(state);
            }
        } else if (state->type == EVENT_WRITE) {
            // Write completed, re-arm read
            add_read_sqe(&ring, state->fd);
            free(state);
        }

        // Mark CQE as processed
        io_uring_cqe_seen(&ring, cqe);
        // Batch submit all queued SQEs
        io_uring_submit(&ring);
    }

    io_uring_queue_exit(&ring);
    close(server_fd);
    return 0;
}

Modern Rust Ecosystem: tokio-uring and glommio

In the Rust ecosystem, developers are leveraging io_uring to build next-generation thread-per-core asynchronous runtimes:

• tokio-uring: Extends Tokio with an io_uring driver, bringing true async file and socket completions to standard Tokio futures.
• glommio: A Thread-per-Core cooperative runtime inspired by the Seastar framework, leveraging io_uring for DMA storage engines and sub-millisecond query pipelines.

// Example: Async File Read in Rust with tokio-uring
use tokio_uring::fs::File;

fn main() {
    tokio_uring::start(async {
        let file = File::open("dataset.parquet").await.unwrap();
        let buf = vec![0u8; 4096];
        
        // Asynchronous DMA read directly into buffer
        let (res, buf) = file.read_at(buf, 0).await;
        let bytes_read = res.unwrap();
        println!("Read {} bytes via io_uring zero-copy!", bytes_read);
    });
}

Benchmark Comparison: epoll vs. io_uring

Summary & When to Migrate

While epoll remains a reliable, universally supported interface for legacy applications and standard web servers, io_uring is the undisputed future of Linux systems programming:

• Use io_uring When: You are building high-throughput storage engines (databases, LSM-trees), 100GbE network proxies, real-time messaging brokers, or need unified asynchronous file + socket I/O without managing thread pool workers.
• Stick with epoll When: You require maximum backwards compatibility across older Linux kernel versions ($< 5.1$), are running inside constrained containers where unprivileged io_uring syscalls are blocked by seccomp profiles, or where I/O throughput is not the primary performance bottleneck.