Interrupt affinity for UDP receive queues

Affinity chooses where notification begins

Each NIC receive queue commonly has a message-signaled interrupt, often MSI-X, that can be directed to selected CPUs. The interrupt schedules NAPI work, which processes descriptors and moves packets into the Linux network stack. Interrupt affinity controls the CPUs allowed to handle that hardware notification.

Good placement reduces competition and cross-socket memory traffic. Bad placement sends a hot queue to a housekeeping CPU, moves packets across NUMA nodes, or makes the ingest application compete with the softirq work that feeds it. Total host CPU can remain low while one receive CPU saturates.

Affinity does not move every later stage automatically. Receive Packet Steering can redirect stack processing. The scheduler can run the application elsewhere. Worker handoffs can move payloads again. Treat IRQ placement as the first CPU decision in a chain and verify observed execution with counters and tracing.

proc interrupts identifies queue activity

/proc/interrupts lists interrupt vectors as rows and CPUs as columns. NIC queue names appear at the right, with forms such as eth0-TxRx-0, eth0-rx-0, ens5-Tx-Rx-3, mlx5_comp0, or driver-specific labels. Counters increase on CPUs that handled each vector.

Sample twice during feed activity and calculate deltas. Static totals may reflect old affinity. The queue carrying one UDP flow should show a clear rate increase. If several names are plausible, correlate with ethtool -S per-queue packet counters and an isolated traffic run.

An IRQ number is not stable inventory. Channel-count changes, driver reloads, reboots, and instance replacement can assign new numbers. Automation should discover by interface and queue name each time, then write the matching affinity and verify it.

affinity_list is safer than hexadecimal masks

/proc/irq/126/smp_affinity_list uses CPU list syntax such as 4, 4-7, or 4,6. /proc/irq/126/smp_affinity uses a hexadecimal bitmask, split into comma-separated 32-bit groups on larger systems. List syntax reduces mistakes when CPU counts exceed one word.

Writing 4 to smp_affinity_list allows CPU 4. Writing 10 to smp_affinity writes hexadecimal 0x10, which also selects CPU 4. Confusing decimal lists with masks can put the IRQ on the wrong core. Always read back both the allowed value and /proc/interrupts deltas.

Some interrupts are managed by the kernel and expose restrictions through effective_affinity_list. A write may fail or the effective set may differ from the requested set. PCI and driver behavior determines which CPUs are actually valid.

irqbalance can overwrite manual placement

irqbalance distributes hardware interrupts according to topology and observed load. On a general-purpose host, it is often useful. On a dedicated market-data receiver, its later decisions can move a carefully placed queue and change latency without an application deployment.

Do not stop irqbalance without replacing its policy for every important device. Configure banned CPUs or per-IRQ hints through the distribution's supported mechanism, or install a service that reapplies interface-specific affinity after device initialization. Options and configuration formats vary by irqbalance version.

Monitor effective placement. A boot-time command can run before the NIC has its final queue layout, and a channel reset can create new IRQs after the service completed. The system is correct only when current interrupt deltas land on the intended CPUs.

RSS decides which queue needs placement

Receive-side scaling hashes flow headers and uses an indirection table to select a hardware queue. ethtool -x eth0 shows that table, and ethtool -n eth0 rx-flow-hash udp4 shows hash fields on supporting drivers. ethtool -l eth0 reports the available and current channel count.

One stable UDP five-tuple usually remains on one queue. Pinning all eight receive IRQs across eight CPUs does not make that feed use eight CPUs. Identify the active queue first. Other flows on the host may still use the remaining queues and need their own placement.

If the source or destination tuple changes, the hash can move the feed. RSS key or table changes can also remap it. A persistent design either constrains the flow with a supported hardware rule or monitors and adapts affinity to the current queue.

NUMA locality sets the first placement boundary

cat /sys/class/net/eth0/device/numa_node returns the NIC's NUMA node when available. lscpu -e=CPU,NODE,SOCKET,CORE maps logical CPUs to topology. An IRQ on the NIC-local node usually reduces inter-socket DMA and cache traffic.

Pin application buffers and receive work near the same node. Linux memory allocation commonly follows first touch, so a pool created by a startup thread on another node may remain remote even after the receiver is pinned. numactl --membind or controlled initialization can make placement explicit.

Virtual NICs may report -1 for unknown NUMA placement. Cloud platforms may map virtual CPUs and network devices through a host topology the guest cannot see. Benchmark candidate CPUs because the visible topology is then incomplete.

Physical cores matter more than logical numbers

Simultaneous multithreading exposes two logical CPUs on one physical core. Placing a hot NIC IRQ on CPU 4 and heavy decoding on its sibling CPU 20 can make both contend for execution resources, caches, and branch machinery. The logical numbers may look separate while the core is shared.

lscpu -e=CPU,CORE,SOCKET,NODE shows sibling relationships. Keep the receive path away from unrelated busy siblings. One option puts the IRQ and ingest thread on separate physical cores in the same NUMA node. Another co-locates them on one core for cache locality and keeps its sibling idle. Measure both.

Core isolation kernel parameters can reduce scheduler noise but affect housekeeping, timers, workqueues, and operational flexibility. Affinity and cgroup cpusets often provide a narrower first step. Do not isolate cores without assigning required kernel work elsewhere.

Pinning the process is insufficient if the runtime creates unpinned helper threads. Inspect taskset -acp for the process thread group or enumerate /proc/24871/task in the concrete example. Allocator background work, asynchronous logging, and runtime workers can still run on the ingest CPU. Give every thread class an intentional cpuset, and verify migrations with perf stat or scheduler tracing.

Hard IRQ and softirq work are related but distinct

The hardware interrupt handler performs limited work and schedules NAPI. Network receive processing often appears as NET_RX softirq time. Under load, it can run in interrupt return context or in a ksoftirqd thread when deferred. top or mpstat reports softirq CPU, while perf and tracing provide finer attribution.

Pinning the IRQ influences where NAPI begins, but Receive Packet Steering can enqueue later protocol processing on another CPU. A congested CPU may also push work into ksoftirqd scheduling behavior. Inspect /proc/net/softnet_stat per CPU. The first field counts processed packets, the second drops, and the third budget exhaustion in hexadecimal.

The goal is not zero softirq time. The goal is bounded receive latency with no local drops and enough application CPU. Moving softirq out of sight does not remove its work.

RPS adds software CPU steering

Receive Packet Steering, or RPS, hashes packets in software and queues processing to CPUs selected in /sys/class/net/eth0/queues/rx-0/rps_cpus. The value is a hexadecimal CPU mask. RPS can distribute protocol-stack work when hardware has too few queues or a virtual device lacks effective RSS.

RPS adds interprocessor interrupts and cache movement. On a NIC with enough well-mapped hardware queues, leaving rps_cpus at zero often avoids redundant steering. For one UDP flow, RPS can preserve flow affinity rather than spread individual packets, depending on hash and configuration.

Receive Flow Steering, or RFS, extends placement toward the CPU where the consuming application runs. net.core.rps_sock_flow_entries sets the global flow table size, and each queue's rps_flow_cnt sets its contribution. RFS can improve cache locality for many flows, but one dedicated feed may need less machinery.

XPS affects outbound traffic, not feed receive

Transmit Packet Steering, or XPS, uses per-queue xps_cpus or xps_rxqs masks to select transmit queues. It can improve locality for acknowledgements or related outbound traffic. The inbound raw UDP feed does not gain receive parallelism from XPS.

This distinction matters because sysfs paths for RPS and XPS look similar. Changing xps_cpus while investigating receive drops will not move the RX queue interrupt or NAPI processing. It may change outbound control traffic and obscure the experiment.

Document each steering layer in one table: RSS flow to RX queue, IRQ to CPU, RPS mask, receiver thread CPU, decoder worker CPUs, and TX queue policy. Most affinity incidents become visible when those mappings are written down.

Co-locating IRQ and receiver has a trade

Putting the queue IRQ and receive thread on the same CPU can keep packet metadata and socket structures cache-local. Once NAPI enqueues a datagram, the application can consume it without a cross-core wakeup. Under moderate load, this often performs well.

The same CPU must schedule both producer and consumer. A long recvmmsg batch or application critical section can delay NAPI. A large NAPI budget can delay the application. At high rate, moving the receiver to a neighboring physical core may let both progress concurrently.

There is no universal placement. Test same-core, adjacent-core, and remote-node layouts. Record NET_RX softirq time, application runtime, migrations, cache misses, socket queue occupancy, and packet age. Pinning without this comparison replaces scheduler variability with a fixed guess.

Reserve housekeeping capacity

Networking hosts still need timers, RCU callbacks, storage interrupts, telemetry, SSH, and kernel workqueues. Assigning every CPU to feed or decoder work creates unpredictable interference when housekeeping has nowhere to run. Reserve one or more cores according to host size and operational load.

Move block-device, management NIC, and high-rate telemetry interrupts away from the feed CPU where possible. /proc/interrupts reveals collisions. Systemd CPUAffinity, cgroup cpusets, and taskset can place applications, while IRQ masks place hardware. Kernel threads may need separate controls.

Avoid per-packet logging on the housekeeping path. At 5,585 packets/sec, verbose logs can saturate storage or a collector and send interrupts back onto carefully isolated CPUs. Metrics should aggregate.

Verify after every topology change

ethtool -L channel changes can recreate IRQs. ethtool -X RSS changes can move flows. Driver upgrades can rename queues. Reboot can change enumeration. Live migration can change virtual topology. Affinity is not a set-once property.

A boot verification should check the interface PCI or virtual identity, NUMA node, queue count, RSS table, discovered IRQ names, requested and effective affinity, RPS masks, process cpuset, and observed interrupt deltas under a feed packet. Fail health checks when the active queue lands outside policy.

Keep the procedure reversible. Store the original affinity lists during an experiment and restore them if tails or CPU pressure worsen. Persistent configuration should be declarative and idempotent, not a collection of commands run by hand.

Measure the full distribution under bursts

The feed mean inter-arrival interval is near 179 microseconds from 5,585 packets/sec, but shreds arrive in bursts. A queue CPU that looks idle at one-second resolution can still be unavailable for a critical 100 microsecond window. Use packet timestamps and scheduler-aware tooling.

Compare p50, p99, p99.9, and maximum receive age, plus IRQ rate, softirq time, softnet drops, UdpRcvbufErrors, socket drops, and decoder queue age. Run at production mean, two-times burst, and with mixed host traffic. Repeat enough times to separate a stable effect from noise.

A successful affinity change removes contention without moving loss or age downstream. If the ordinary kernel path meets the budget with one local queue and one ingest core, more steering is unnecessary. Simpler placement is easier to keep correct through upgrades.