SO_REUSEPORT for high-rate UDP receivers

One port can have multiple sockets

SO_REUSEPORT allows multiple sockets to bind the same local IP address and port when they satisfy Linux's reuse rules. For UDP, the kernel selects one socket from the reuseport group for each incoming datagram. This can distribute work across receiver threads or processes without one shared receive lock and one user-space dispatcher.

Every member must set SO_REUSEPORT before bind. Binding one ordinary socket first and attempting to add a reuseport socket later fails. Processes normally need compatible effective user identities, a protection against an unrelated process joining a listener group. Exact checks have evolved across kernel versions.

Group membership is local to a network namespace and compatible bind address. A socket bound to one specific local address does not form the same delivery target as every wildcard binding in all cases. Standardize address family, IPV6_V6ONLY policy, local address, port, and socket options across workers. Mixed bindings are difficult to diagnose and can produce host-specific delivery choices.

The option changes local delivery, not network delivery. The NIC still receives the packet, RSS still selects a hardware queue, and the kernel still performs IP and UDP lookup. SO_REUSEPORT decides which eligible socket gets the datagram. It does not duplicate the datagram to every socket.

SO_REUSEADDR is a different option

SO_REUSEADDR primarily changes address-binding rules, including cases involving wildcard and specific addresses and reuse after prior bindings. Its UDP behavior differs across operating systems. It should not be treated as a portable load-balancing switch.

SO_REUSEPORT is the explicit Linux mechanism for a group of listeners on the same endpoint. Production code commonly sets both when its binding and restart semantics require them, but each should have a documented reason. Setting a collection of socket options copied from a sample creates surprising security and delivery behavior.

Test startup ordering. If one worker fails to set SO_REUSEPORT, it may either fail its bind or become a separate binding obstacle. The supervisor should treat a partial group as unhealthy rather than running with an unexpected number of queues.

Default selection uses a packet hash

Linux's default reuseport selection hashes packet and socket information to choose a group member. This aims to keep a flow on one socket while distributing distinct flows. Flow affinity is useful for connection state and cache locality. It also limits scaling for a one-flow UDP feed.

A flow is commonly identified by source address, source port, destination address, destination port, and protocol. If every shred arrives from one fixed source tuple to one destination tuple, the hash remains stable and one reuseport socket can receive nearly every packet. Starting eight workers does not guarantee an eight-way split.

Measure per-socket packet counts. Do not infer distribution from thread CPU alone. One worker may do decoding for packets dispatched by another, or the scheduler may move tasks. Add an rx_datagrams counter per listener index and compare it with socket drops and queue placement.

One raw feed may be one flow

shredstream.sh sends from the published source IP 64.130.40.90 to the customer's verified destination. If the source port and destination port are stable, the resulting five-tuple offers no natural flow diversity. Default SO_REUSEPORT hashing then preserves affinity rather than spreading individual shreds.

This is not a failure in reuseport. It is the intended behavior for ordinary flows. Distributing packets from one flow requires an explicit policy, multiple source or destination tuples, NIC-level steering that changes queue behavior, or a user-space handoff after one receive socket.

At the measured 5,585 packets/sec, one optimized native receive loop should have ample packet-rate capacity on modern server hardware. The harder work is usually recovery and decoding. One ingest socket can hand off by slot, fec set, or hash to workers while preserving the protocol grouping needed downstream.

BPF can select a reuseport socket

Linux supports attaching classic or extended BPF programs to a reuseport group with SO_ATTACH_REUSEPORT_CBPF or SO_ATTACH_REUSEPORT_EBPF. The program returns an index into the group. A valid result selects that socket, while an invalid result falls back to the normal reuseport selection behavior.

An eBPF selector can inspect available packet and context data and implement a policy aligned with application semantics. For example, it could spread by bytes that identify a slot or fec group, provided offsets, variants, and validation are safe at that hook. A malformed or changing wire layout can silently destroy distribution quality.

The program adds operational complexity. Socket ordering matters because group indices can change when sockets close, with the last socket moved into a vacated position. Load, replacement, and rollout behavior need tests. A BPF program that parses protocol bytes is part of the decoder compatibility surface.

Socket-per-CPU design reduces shared state

A common layout creates one reuseport socket per selected CPU, pins each receive thread, and gives each thread its own buffer pool, counters, and handoff queue. Avoiding a shared receive queue can reduce lock contention and cache-line movement at high rates.

The design works best when network queue and socket selection agree with CPU placement. SO_INCOMING_CPU can set or inspect CPU preference in relevant Linux paths. SO_INCOMING_NAPI_ID reports the NAPI identifier associated with received traffic on supported kernels. Those signals help validate locality.

Separate sockets also mean separate SO_RCVBUF allocations and drop counters. Four workers requesting 4 MiB each have four receive-memory budgets, reported doubled by Linux. Host-wide UDP memory limits and total RAM planning must account for the group, not one socket.

Receive-buffer imbalance is a useful diagnostic. One socket with rising r and d fields while its peers remain empty indicates hash or BPF skew, not a host-wide memory shortage. Export the effective rb value from every worker at startup and sample occupancy during bursts. A group is healthy only when the intended distribution and every member's drain rate remain inside policy.

RSS and reuseport solve different selections

Receive-side scaling selects a NIC receive queue, usually through a hash over packet headers. SO_REUSEPORT selects an application socket after the packet enters the stack. Their hash functions and indirection are not automatically coordinated. A packet can land on CPU 2's NIC queue and be delivered to a socket whose thread runs on CPU 6.

ethtool -x eth0 shows the RSS indirection table and hash key on supported drivers. ethtool -n eth0 rx-flow-hash udp4 shows which fields participate in the UDP IPv4 hash where implemented. /proc/interrupts maps queue IRQ activity to CPUs. The application must expose which socket received each flow.

One five-tuple also tends to select one RSS queue. Reuseport distribution in the stack cannot undo the hardware queue's interrupt and NAPI concentration. It can distribute later application work, but the first receive path remains on one queue unless the hardware or flow layout changes.

Ordering changes when packets are spread

One socket preserves the kernel's enqueue order for packets delivered to that socket. Multiple sockets processed by multiple threads produce no single application order. Even if network arrival is ordered, scheduler and batch differences can make a later packet finish parsing first.

UDP already permits network reordering, so a correct shred receiver must tolerate it. Reuseport can increase the observed reorder window and downstream concurrency. Sequence handling should be explicit rather than relying on one-thread timing.

Protocol-aware steering can improve locality by sending related shreds to the same worker. Round-robin packet distribution maximizes balance but may force shared fec-set state or cross-core transfers. Hashing slot and fec-set identity can keep recovery state local, although skewed sets may temporarily imbalance workers.

Worker lifecycle affects packet mapping

Adding or removing a socket changes the reuseport group and can remap flows. During a rolling restart, one stable flow may move to a new process. UDP has no connection handshake to announce that transition. State held only in the old worker can become unavailable while later packets arrive at the new one.

For stateless packet validation this is minor. For fec recovery and slot assembly it can split a live group across process generations. Use shared or transferable state, drain at protocol boundaries, or accept a bounded recovery loss during deployment.

Supervisors should start the replacement group, verify all sockets and BPF attachment, then switch traffic through a controlled mechanism. Joining new sockets directly into a live group may be valid, but the mapping consequence must be part of the rollout design.

Observability becomes per socket and per queue

ss -u -a -n -m -p lists UDP sockets, process information, and skmem details when permissions allow. Multiple rows with local port 9000 confirm multiple kernel sockets. Each row has its own receive allocation and drops field. /proc/net/udp also lists each socket and its inode.

Application metrics should label a small fixed worker index, CPU, and listener group generation. Record packets, bytes, socket receive failures, user-queue rejects, batches, decode work, and maximum packet age. Avoid labels containing arbitrary ports or process IDs if they create unbounded time series.

At the NIC layer, collect per-queue counters from ethtool -S where the driver exposes them. A balanced reuseport group with all NIC traffic on one queue may still suffer queue-level pressure. A balanced RSS table with all application delivery on one socket exposes the opposite mismatch.

Failure isolation can justify reuseport

Throughput is not the only reason for multiple sockets. Separate processes can isolate runtime pauses, allocator failures, and decoder crashes. One worker can restart while others continue receiving different flows. Per-socket queues prevent one worker from locking a shared user-space ring.

Isolation is incomplete for a single default-hashed flow because only one socket may carry it. If that worker stalls, its socket fills while idle group members remain empty. BPF distribution or multiple publisher flows are required to make the spare capacity relevant.

Multiple processes also complicate deduplication, metrics aggregation, shutdown, and recovery state. At 5,585 packets/sec, a single ingest process with well-isolated downstream workers may be the clearer and safer architecture. Use reuseport when measured contention or failure-domain requirements support it.

Security and binding rules matter

SO_REUSEPORT allows a process to receive traffic intended for the shared endpoint, so Linux applies identity checks to group membership. Containers, user namespaces, ambient capabilities, and process supervisors can make the effective identity less obvious. Verify with the deployed security model.

Bind to the intended destination address rather than 0.0.0.0 when interface scope matters. Apply firewall rules restricting source IP 64.130.40.90 and the chosen UDP destination port. Source filtering narrows exposure but does not authenticate shreds. Protocol signatures and validation remain required.

A BPF selector must handle short and malformed packets safely. It runs before the full application validator and should default to a harmless distribution result rather than reading beyond available data. Load tests need adversarial packet sizes as well as the expected 1,216 byte mean.

The simplest scaling point is often after receive

One receive thread using recvmmsg can ingest many times 5,585 small datagrams/sec on suitable hardware. It can timestamp, validate length, identify a routing key, and place pointers into per-worker rings. The expensive work then scales across cores without complex socket steering.

This creates one handoff and a potential single ingest failure domain. It also creates one place for gap tracking, source validation, capture sampling, and overload policy. The design is easier to reason about than BPF parsing and dynamic reuseport indices.

Benchmark both architectures with the same burst and pause tests. Compare p99.9 receive age, cache misses, user-ring rejects, socket drops, and recovery outcome. Choose the smaller system that meets the target. SO_REUSEPORT is a precise tool, not a required badge for a high-performance receiver.