A service can replace epoll with io_uring, cut syscall counts dramatically, and still lose its p99 latency target. The usual cause is not that io_uring is slow. It is that io_uring performance tuning was treated as an API migration instead of a queueing, CPU-ownership, and memory-lifetime problem.
At high concurrency, the ring is only one stage in a request path that also includes NIC queues, socket buffers, scheduler decisions, page cache behavior, filesystem workers, and application handoff. A million operations per second spread across too many rings, poorly sized batches, or a completion path that cannot keep up will produce the same familiar symptoms: rising tail latency, excess CPU, completion-queue overflow, and throughput that plateaus well below device capacity.
io_uring Performance Tuning Starts With the I/O Path
Before changing ring flags or queue depth, identify what actually happens after submission. Network receive, buffered file I/O, direct I/O, and block I/O have different execution paths and different reasons to use io_uring.
For a network server, the primary win is often reducing readiness polling and syscall transitions while keeping request handling close to the CPU that owns the connection. For buffered file reads, the page cache may dominate the result. If an operation cannot complete through the fast path, the kernel can use io-wq worker threads. That fallback preserves asynchronous semantics, but it also means a workload expected to be event-driven may consume threads and introduce scheduler latency.
Start with a production-shaped benchmark, not a microbenchmark that submits reads from a hot cache. Capture throughput, p50 and p99.9 completion latency, cycles per operation, context switches, softirq load, ring occupancy, and the rate of operations handled by io-wq. A useful baseline separates time spent waiting for data from time spent after the completion is visible to userspace.
If the system is already saturated on a NIC receive queue, storage device, or a mutex in the application, a faster submission interface will not create capacity. It can make the next bottleneck more obvious.
Size Queue Depth as a Control Loop
Queue depth should represent useful work in flight, not a large number selected because a storage benchmark used it. The right depth depends on service time, target throughput, the number of shards, and the amount of latency variance the system can tolerate.
A ring that is too shallow starves the device or network path. A ring that is too deep converts transient backpressure into a large tail-latency queue. For request-response services, that is usually the more expensive failure mode. Requests that sit in an oversized submission backlog consume memory, retain buffers, extend cancellation windows, and can time out after the work has already become inevitable.
Tune from observed occupancy. If the submission queue is routinely empty while the device is underutilized, increase the amount of work staged per owner. If completions arrive in bursts and the completion queue approaches capacity, first verify that the application is draining completions quickly enough. Increasing ring size can hide a slow completion consumer, but it does not fix it.
Batching helps because one enter operation can submit multiple SQEs and retrieve multiple CQEs. It hurts when batching waits too long for a full batch. For latency-sensitive traffic, use a bounded batching policy: submit when the batch reaches a modest threshold or when a short deadline expires. The threshold should be measured against traffic shape, not copied from a bulk-storage workload.
Keep Submission and Completion Ownership Local
Cross-core ring use is a frequent source of disappointing results. If one set of threads submits work and another set drains completions, the shared ring metadata can become a cache-line ping-pong point. The overhead is small at low rates and punishing at several hundred thousand operations per second.
Prefer a sharded design: one ring per event-loop owner or CPU-affine worker, with the same worker submitting and consuming whenever possible. Pinning is not mandatory in every deployment, but predictable CPU ownership is valuable when tail latency matters. It aligns cache locality, connection ownership, and completion handling.
Where the deployment supports it, setup options such as single-issuer semantics can reduce synchronization overhead by expressing what the application already guarantees. Do not claim that guarantee casually. A future refactor that allows multiple submitters can turn an optimization assumption into a correctness failure.
Remove Hidden Work Before Adding Kernel Threads
Registered files and registered buffers can remove repeated lookup and pinning work from a hot path. They are especially useful when a service operates on a stable set of sockets or files and reuses a bounded memory pool. Fixed-buffer I/O also makes buffer ownership explicit, which is operationally useful when diagnosing leaks and stalled requests.
The trade-off is lifecycle complexity. A registered buffer must remain valid until the kernel has completed every operation that references it. Reusing it early creates the kind of intermittent corruption that survives staging and appears only under overload. Build buffer pools around completion ownership, not around optimistic assumptions about when a submission has left userspace.
Buffer rings and provided buffers are often a better fit for high-rate receive paths than allocating a buffer per request. Replenish them aggressively, but treat depletion as a first-class saturation signal. When no receive buffers are available, the correct response is usually to apply admission control or shed work upstream, not to grow the pool without bound.
SQPOLL can eliminate submission-side system calls by running a kernel polling thread for the submission queue. It is useful when a dedicated, busy ring has enough sustained traffic to justify an always-available CPU. It is a poor default for sparse traffic, heavily oversubscribed hosts, or fleets where idle polling threads turn a latency experiment into a power and cost problem.
IOPOLL has an even narrower use case. It is intended for compatible direct-I/O block devices and relies on polling for completion. It can improve latency consistency for storage-heavy systems, but it consumes CPU while waiting and does not apply to ordinary buffered file I/O or general network traffic.
Match io_uring Features to the Workload
Network servers should evaluate multishot receive and accept operations where their kernel and application lifecycle support them. These features can reduce resubmission churn, but they require careful handling of terminal completions, buffer selection, connection teardown, and backpressure. A multishot operation is not a license to stop modeling per-connection limits.
For storage, distinguish buffered reads from direct I/O. Buffered reads may be fast because data is resident in the page cache, while cache misses can involve filesystem and worker-thread behavior that changes the latency profile entirely. Direct I/O avoids page-cache effects but imposes alignment requirements and shifts pressure toward the block device and application buffer management.
Linked operations are valuable when ordering matters, such as a write followed by a dependent fsync. They are not a substitute for understanding failure propagation. If one request in a chain fails or is canceled, downstream behavior must be designed and tested explicitly. At scale, ambiguous cleanup logic is more damaging than an extra syscall.
Instrument Completions, Not Just Submissions
Submission rate is an attractive metric because it rises immediately after an optimization. Completion rate and completion latency tell you whether the system did useful work. Track SQ occupancy, CQ occupancy, CQ overflow events, failed submissions, cancellation results, short reads or writes, and time spent between a CQE becoming available and the application consuming it.
Pair application metrics with kernel evidence. Scheduler traces expose whether io-wq or application workers are waiting on CPU. Block-layer metrics show whether queueing occurs below the filesystem. Network counters reveal drops and socket pressure that a ring-level dashboard cannot see. CPU profiles frequently uncover the real issue: expensive protocol parsing, allocator contention, logging, or a lock protecting a supposedly asynchronous path.
Test overload deliberately. Cap available CPUs, inject slow storage, exhaust provided buffers, force cancellation storms, and verify behavior when the completion queue is pressured. The service should degrade through bounded queues and explicit rejection, not through unbounded memory retention or a growing population of blocked worker threads.
The best io_uring deployment is rarely the one with the most flags enabled. It is the one whose ring ownership, queue depth, buffer lifetime, and backpressure policy remain understandable at 3 a.m. while the system is operating near its limits.


Leave a Reply