Your Bot Is Dropping Shreds: UDP Buffers, Backpressure and Silent Loss

Why this fails silently

TCP has a feedback loop. If your application stops reading, the receive window closes, the sender slows down, and eventually something times out and complains. You find out. UDP has none of that. A shred provider forwards datagrams at line rate and never asks whether you are keeping up. If you are not, packets are discarded - at your NIC, in your kernel, or at your socket - and nothing anywhere in the path raises an error. Your recv() loop keeps returning packets. They are just not all the packets. This is not a flaw in the design; it is the design. Retransmitting a shred that arrives after the slot is decided would be worse than useless, so the protocol does not try. It ships redundancy instead, which we will come back to. The practical consequence is that a shred feed cannot tell you it is broken. You have to look. The companion article explains the protocol trade-off; this page focuses on receiver operations.

Symptom: I see gaps in slot or shred indexes

Start here, because it narrows fastest. Packets can die at four places, and they are diagnosed differently.

wire → [1] NIC ring buffer     → [2] kernel softirq backlog     → [3] socket receive buffer     → [4] your application's own queue

Layers 1 and 2 are host-wide. Layer 3 is per socket. Layer 4 is your code. Work through them in order - the counter that is moving tells you where to spend your afternoon.

1. The socket buffer - check this first, it is usually this

Per-socket drops, keyed by port:

# last column is 'drops'cat /proc/net/udp | awk 'NR==1 || $2 ~ /:4E20$/'   # 0x4E20 = port 20000

Or the system-wide totals:

cat /proc/net/snmp | grep -A1 '^Udp:'

The fields are InDatagrams NoPorts InErrors OutDatagrams RcvbufErrors SndbufErrors InCsumErrors IgnoredMulti MemErrors. RcvbufErrors is the one that means "your receive buffer overflowed." If it is rising, layer 3 is your problem. netstat -su prints the same thing more readably as "receive buffer errors", and ss -ulmn shows per-socket memory with a d<n> drop count in the skmem field.

2. The kernel backlog

# column 2 is 'dropped', column 3 is 'time_squeeze', one row per CPUcat /proc/net/softnet_stat

The second field indicates pre-socket discard when the input queue fills; a rising third field indicates softirq budget pressure. Either result points to host-wide load rather than only a slow consumer.

3. The NIC

ethtool -S eth0 | grep -Ei 'drop|miss|err|fifo|nobuf'

When rx_missed_errors or rx_no_buffer_count rises, the NIC ring filled before the kernel could drain it. Confirm that this layer is responsible before enlarging the ring; on a dedicated receiver, application stalls are often the more likely explanation.

4. Your own code

If every counter above is zero and you are still missing shreds, the loss is inside your process - a bounded channel that drops when full, a parser that blocks, a select loop that does real work between reads. The kernel handed you the packets and you lost them. Only your own instrumentation can see this, which is why you should have a counter on every queue that can drop.

Symptom: it only happens under load

That is layer 3, and the fix is arithmetic.

The receive buffer is a time budget, not a size

The only useful way to think about SO_RCVBUF is: how long can my application stall before the kernel starts discarding? headroom_ms = rcvbuf_bytes / feed_bytes_per_sec × 1000

Measure your feed rate first - do not guess it:

sudo timeout 60 tcpdump -ni eth0 -q "udp dst port 20000" -w /tmp/s.pcapcapinfos -d /tmp/s.pcap    # gives bytes; divide by 60

Say it comes out around 6.8 MB/s. Then:

Thirty-one milliseconds. That is the default. A garbage collection pause, a page fault, a log flush to a busy disk, one slow write() - any of these blows through it, and every packet that arrives during the stall is gone with no error. This is why the problem shows up under load and disappears when you go looking for it.

The two gotchas that waste everyone's afternoon

The kernel returns double what you set. Ask for 1 MB and getsockopt reports 2,000,000 - the kernel adds an equal allowance for its own bookkeeping. This is normal and you have not misread anything. Linux silently caps the request at its configured ceiling. Read the returned value after setting it; a successful call does not prove the requested capacity was granted. Here is that happening:

>>> s.setsockopt(SOL_SOCKET, SO_RCVBUF, 64 << 20)   # ask for 64 MiB>>> s.getsockopt(SOL_SOCKET, SO_RCVBUF)8388608                                             # got 8 MiB. no error.

Always read back what you actually got, and log it at startup. A receiver that assumes it has 64 MiB while holding 8 is the exact shape of a mystery incident.

Fixing it

# raise the ceiling, then re-run your receiversudo sysctl -w net.core.rmem_max=134217728sudo sysctl -w net.core.netdev_max_backlog=10000

# persistecho 'net.core.rmem_max = 134217728'      | sudo tee -a /etc/sysctl.d/99-shreds.confecho 'net.core.netdev_max_backlog = 10000' | sudo tee -a /etc/sysctl.d/99-shreds.conf

Then in your receiver, request it and verify:

s.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 128 << 20)actual = s.getsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF)log.info("shred socket rcvbuf: %d bytes (%.1f s headroom at %.1f MB/s)",         actual, actual / feed_rate, feed_rate / 1e6)

A bigger buffer buys time, it does not buy throughput. If your consumer is slower than the feed on average, no buffer size saves you - it only changes how long you take to fall over. Which brings us to the real fix.

Symptom: the buffer is huge and I still drop

Then the buffer was never the problem. Your read path is too slow, and the architecture has to change.

Read on a thread that does nothing else

One thread whose entire job is recv() into a preallocated ring buffer, and nothing else - no parsing, no allocation per packet, no logging, no locks held across a read. Everything downstream happens on another thread.

Use a bounded queue and drop deliberately

When the ring is full you are going to lose data; the only question is whether you choose which. Dropping the oldest shred is almost always right - stale shreds are worthless in this business - and dropping on purpose with a counter is infinitely better than the kernel dropping the newest silently.

Measure queue age, not queue depth

Depth tells you how much work is waiting. Age tells you how stale the front of it is, which is the number that determines whether the work is still worth doing. Alert on age.

Decide locally whether stale work is useful

Apply an age limit at the consumer. Once a shred is behind the slot you can act on, discard it so stale work cannot delay the next item.

Turn off what you do not need

Reverse DNS, per-packet logging, per-packet allocation, recvfrom when recv will do. recvmmsg batches syscalls and helps meaningfully at these rates.

Symptom: I lose whole slots, not scattered shreds

That is a different failure, and FEC is why it matters. FEC sets provide recovery: current senders commonly emit 32 data and 32 parity shreds. SIMD-0317 discusses making that arrangement enforceable, but the proposal is not active. Reed-Solomon recovery can rebuild a set from any 32 of the 64. That is a lot of tolerance. You can lose half of a set and lose nothing. But it is 32 per set, and packet loss is bursty rather than uniform. A buffer overflow does not politely discard every other packet - it discards everything that arrives during the stall, which is very likely to be more than 32 consecutive shreds from the same set. So:

• Scattered loss: FEC absorbs it. You may never notice.
• Bursty loss: takes out whole FEC sets. You lose entire slots.

Losing whole slots is the signature of a stall, not of a bad network. If you see it, look at your buffer and your consumer before you look at your provider. Conversely, only-on-A style asymmetry spread thinly across many slots - the kind shredbench reports when comparing two feeds - is more likely to be genuine upstream loss. Under merkle authentication, data shreds are 1,203 bytes and coding shreds 1,228, so a larger datagram should raise an alarm. Measure after reconstruction; pre-recovery gaps are expected and are not, by themselves, application loss. Full wire format on our shred format reference.

What to monitor, permanently

A UDP feed gives you no session, so every signal is local. These are the seven that matter:

Packet age is the one that catches an outage. Everything else diagnoses; that one pages. If you build only one of these, build that. And when you migrate providers, remember the alerts that watched the old feed's session state are about to go permanently green rather than red. Green is worse. Replace them.

The five-minute checklist

# 1. Is the per-port drop counter moving?cat /proc/net/udp # 2. Are receive buffer errors climbing?cat /proc/net/snmp | grep -A1 Udp: # 3. Is the kernel backlog dropping packets?cat /proc/net/softnet_stat # 4. Is the NIC missing frames?ethtool -S eth0 | grep -i drop # 5. Is SO_RCVBUF clamped below the requested value?# Log the actual value returned by getsockopt at startup. # 6. Raise the receive-buffer ceiling and re-run.sudo sysctl -w net.core.rmem_max=134217728 # 7. Still dropping? Move parsing off the read thread.