NIC tuning for high-rate UDP

The NIC owns the first receive queues

Packets reach hardware before Linux can place them in a socket buffer. The NIC validates frames, may verify checksums, classifies traffic, selects a receive queue, writes packet data into host memory, and notifies a CPU. Ring descriptors, DMA, interrupt moderation, receive-side scaling, and driver behavior determine whether that handoff is timely.

Socket tuning cannot recover a frame dropped because the receive ring had no descriptor. Application batching cannot repair an interrupt routed to an overloaded CPU. Start UDP tuning at the full receive path, while changing only the layer supported by current evidence.

The measured shred feed is 54.3 Mbps and 5,585 packets/sec. The bit rate is small beside a 1 Gbps port, but the packet rate still creates per-packet work. More important, averages hide bursts and CPU pauses. A NIC configuration should absorb the tested burst while keeping queue residence within the strategy's latency budget.

Inventory hardware, driver, and link first

ethtool -i eth0 reports the driver, version, firmware, bus information, and feature support indicators. ethtool eth0 reports speed, duplex, autonegotiation, port, and link state. lspci -vv identifies PCIe link width and speed for physical devices. A fast NIC in a constrained PCIe slot can become a throughput problem at higher rates.

Virtual NICs expose a different contract. ENA, virtio-net, gVNIC, and hypervisor devices may not support ring or coalescing changes through ethtool, and host-side queues remain outside the guest's view. Cloud instance network limits and packet-per-second caps matter more than the apparent virtual link speed.

Record the inventory with every benchmark. Driver and firmware upgrades can change defaults, available counters, interrupt behavior, and offload correctness. A tuning result without its versions is difficult to reproduce.

Ring depth absorbs a bounded service pause

The receive ring contains descriptors that point to buffers available for incoming frames. ethtool -g eth0 displays supported maximum and current RX and TX ring values when implemented. ethtool -G eth0 rx 4096 requests a 4,096 descriptor RX ring.

A deeper ring can absorb a microburst or a short delay before the driver polls it. At 5,585 packets/sec, 4,096 packet positions represent about 733 ms at the mean rate in a naive count. Real capacity and age differ because queues, bursts, descriptor use, and NAPI service proceed concurrently. Holding hundreds of milliseconds would be unacceptable for many strategies.

Larger rings consume memory and expand possible queueing delay and cache footprint. If a receive CPU cannot keep up with steady traffic, every finite ring fills. Increase depth only to cover a defined transient and verify p99.9 arrival-to-user latency.

Driver drop counters identify ring pressure

ethtool -S eth0 prints device and driver statistics. Names are not standardized. Common receive-loss patterns include rx_missed_errors, rx_no_buffer_count, rx_discards, rx_dropped, rx_queue_0_drops, and rx_out_of_buffer. One driver may expose several, another none.

Map counters through vendor documentation and a controlled overload. Take a baseline immediately before a test, then compute deltas. A field that never changes when the receive thread is deliberately stalled may not cover the queue you think it does.

ip -s link show dev eth0 provides generic RX errors, dropped, overruns, frame errors, and carrier information. These totals also need deltas. Clean NIC counters do not exclude later softnet, socket, or application loss.

Interrupt moderation trades latency for interrupt rate

Interrupt coalescing delays or batches NIC notifications. ethtool -c eth0 shows supported values such as rx-usecs, rx-frames, tx-usecs, tx-frames, adaptive-rx, and adaptive-tx. ethtool -C eth0 rx-usecs 4 rx-frames 1 requests a receive interrupt after a short time threshold or frame threshold on devices that interpret those fields conventionally.

Lower coalescing can reduce latency to the first packet in a burst and increase interrupts. Higher coalescing reduces CPU notification overhead while adding wait time. Adaptive moderation changes settings with traffic, which may help throughput but can make tail latency workload dependent.

Test values rather than assuming zero is best. At sustained traffic, NAPI may already remain active and interrupts become less important. At sparse traffic, a 16 microsecond coalescing delay can appear directly in p50. One queue dedicated to the feed may use lower moderation than bulk queues if the driver supports per-queue configuration.

RSS maps flows to hardware queues

Receive-side scaling hashes packet header fields and uses an indirection table to select an RX queue. ethtool -l eth0 shows configured combined or separate channels. ethtool -x eth0 prints the RSS indirection table on supported devices. ethtool -n eth0 rx-flow-hash udp4 reports which fields participate in the UDP IPv4 hash.

One stable UDP five-tuple generally maps to one queue. Increasing the channel count from four to eight does not split packets within that flow. It can still isolate unrelated traffic by distributing other flows. If the host receives feeds on multiple destination ports, RSS can spread them according to the configured hash.

ethtool -X changes the indirection table, and ethtool -N can add receive classification rules on supporting drivers. These are high-impact changes. Record the existing table and rules, and verify after link reset because virtual or managed devices may rewrite them.

Queue count should match active work

ethtool -L eth0 combined 4 requests four combined channels where supported. More queues create more IRQs, rings, and possible parallelism. They also consume memory and make affinity more complex. The correct count follows flow diversity and CPU topology, not the maximum displayed by the device.

A one-flow receiver may need one low-latency RX queue and several queues for host background traffic. Eight idle feed queues offer no throughput gain. Conversely, many distinct feeds on one queue can saturate one NAPI poller while other cores are idle.

Changes to channel count can reset RSS mappings, IRQ numbers, and interface state. Reapply and verify affinity after ethtool -L. Automate discovery by queue name rather than hard-coding an IRQ number that changes after reboot.

IRQ affinity determines which CPU starts work

/proc/interrupts lists per-CPU interrupt counts and device queue names. /proc/irq/126/smp_affinity_list shows the allowed CPUs for IRQ 126. Writing 4 pins that example IRQ to CPU 4, subject to driver and irqbalance behavior.

Place the feed queue on an isolated or lightly loaded CPU in the NIC's NUMA node. Pin the receive thread either to that CPU or a nearby sibling according to measured NAPI and application contention. Putting both on one CPU maximizes locality but makes them compete. Splitting them across nearby cores can improve scheduling at a small cache cost.

irqbalance may move managed IRQs. Configure its banned CPUs or policy, disable it only when the host has a complete replacement policy, and verify placement continuously. CPU hotplug and interface resets can also change effective affinity.

Offloads reduce work and alter observations

ethtool -k eth0 displays checksum, segmentation, receive-hashing, GRO, LRO, and other offload states. Receive checksum offload can avoid software checksum computation. Generic receive offload can combine compatible packet processing. Large receive offload is more commonly associated with TCP and driver-specific behavior.

UDP GRO and UDP segmentation offload support exists in modern Linux paths, but behavior depends on driver, kernel, and socket use. Offloads can improve throughput while changing capture appearance and batching latency. A receiver must preserve individual datagram semantics.

Do not disable every offload because tcpdump shows an apparently bad transmit checksum. The checksum may be completed by hardware after the capture hook. Change one feature during an isolated test, record the old state, compare correctness and latency, then restore it if no benefit is measured.

Flow control can move loss into latency

Ethernet pause frames allow a receiver to ask a link partner to pause transmission for a priority or entire link, depending on link-layer features. ethtool -a eth0 reports generic pause parameters, and ethtool -A changes them where supported. Priority Flow Control belongs to data-center bridging configuration and requires network-wide design.

Pause can prevent local drops during a short burst. It can also create head-of-line blocking for unrelated traffic sharing the paused class or link. A congested receiver can propagate delay upstream. That behavior conflicts with the isolation usually desired for one-way market data.

Do not enable pause as an isolated host tweak. Switch configuration, queue mapping, traffic classes, and deadlock risks matter. For a 54.3 Mbps feed on a correctly provisioned link, local capacity and queue placement are clearer controls.

NUMA placement affects DMA and memory access

A physical NIC attaches to a PCIe root complex associated with a NUMA node. cat /sys/class/net/eth0/device/numa_node reports the node when Linux knows it. lscpu -e=CPU,NODE,SOCKET,CORE maps CPUs to nodes. numactl --hardware shows memory topology.

Place receive threads and UMEM or ordinary application buffers on the NIC-local node when practical. Remote DMA and cross-socket processing add interconnect traffic and latency variance. Automatic memory allocation follows first-touch behavior for many mappings, so initialize pools from the intended CPU or bind memory explicitly after testing.

NUMA locality is not absolute. A decoder may need state owned on another node, and concentrating every network process near one NIC can cause contention. Measure remote memory events and end-to-end latency before moving all work.

Power management can dominate sparse latency

CPU C-states and frequency scaling affect how quickly a core responds to an interrupt. PCIe Active State Power Management can add link wake latency in some systems. ethtool --show-eee eth0 reports Energy Efficient Ethernet state when supported. Server firmware exposes further power policies.

Low-latency hosts often select a performance CPU governor and constrain deep sleep states, but the energy and thermal cost is substantial. Modern hardware behavior differs, and disabling every power feature can reduce turbo headroom. Tune one policy with hardware telemetry.

Sparse packet tests are essential because a continuously busy benchmark keeps cores and links awake. The first packet after idle may show the latency the trading system cares about at slot boundaries or during feed pauses.

A change matrix prevents superstition

Capture a baseline with current ring, coalescing, channels, RSS, offloads, affinity, link state, driver version, and firmware. Change one dimension, replay the same numbered packet stream, and compare loss plus p50 through maximum latency. Retain CPU and IRQ measurements.

Useful experiments include RX ring 512 versus 2048 versus 4096, rx-usecs 0 versus 4 versus 16, adaptive-rx on versus off, and one versus four channels. Not every device accepts every combination. A rejected ethtool command is preferable to a silent assumption.

Select a setting with margin, then test mixed host traffic, burst rate, receive-thread pauses, interface reset, and reboot persistence. A benchmark-only command that disappears on restart is not production tuning.

The ordinary kernel path should remain observable

Keep driver counters, /proc/net/softnet_stat, nstat UDP counters, /proc/net/udp drops, and application sequence metrics in one dashboard. The order gives a localization chain from NIC to strategy. Sample queue-specific counters where the driver provides them.

After a NIC change, success means more than zero rx_missed_errors. Socket drops, packet age, CPU saturation, and application gaps must also remain inside bounds. A deeper ring may move loss from hardware to a later queue. Lower coalescing may move CPU pressure into softnet budget exhaustion.

NIC tuning is complete when the system has a tested configuration, a reason for each nondefault value, persistent automation, monitoring, and a rollback. Unexplained maximum values are not evidence of care.