How to measure one-way UDP latency

One-way latency is a subtraction

One-way latency is the receiver timestamp minus the sender timestamp for the same packet. The subtraction is meaningful only when both timestamps represent defined points, share a time scale, and have bounded offset error. A nanosecond API does not produce nanosecond accuracy by itself.

For packet i:

one_way_i = receive_time_i - send_time_i

If the receiver clock is 80 microseconds ahead of the sender, every reported latency is inflated by 80 microseconds. If offset drifts during the run, the distribution changes even when the network does not. Negative latency usually indicates clock error, timestamp-point mismatch, or packet identity error, not faster-than-time delivery.

Document the result as a measurement interval plus clock uncertainty. A p99 of 310 microseconds with a possible 100 microsecond clock offset is not a 310 microsecond network claim. The uncertainty is too large for that precision.

Define the two timestamp points first

Sender application time is taken before or after sendmsg in user space. Software transmit time can be taken in the kernel when the skb enters the transmit path. Hardware transmit time is taken by the NIC near actual egress. These points include different amounts of application, scheduler, queue, driver, and link delay.

Receiver hardware time is taken by the NIC near ingress. Kernel software receive time is taken later in the network stack. Application time is taken after recvmsg or after parsing. Subtracting hardware egress from hardware ingress estimates network transit between NICs. Subtracting application send from application receive measures a broader pipeline.

Neither is universally correct. A provider-to-strategy claim may need application-to-application time. A network tuning experiment may need hardware-to-hardware time. Give each metric a name that includes its endpoints, such as sender_post_sendmsg_to_receiver_post_recvmmsg.

Packet identity must survive reordering

Match timestamps by a unique packet identity, never by row order. UDP can reorder, duplicate, and drop. A test datagram should carry a run ID and monotonically increasing sequence number. If the sender embeds its timestamp, that timestamp belongs beside the same identity under a versioned message layout.

For raw shreds, slot, shred index, type, and recovery-set fields can help identify protocol data, but duplicate copies from different sources may share those fields. A measurement wrapper or capture hash may be needed to distinguish physical sends. Do not modify production shred bytes unless the test path explicitly uses a separate envelope.

Deduplicate before calculating quantiles. Decide whether the first or earliest receive copy represents latency. Record duplicate latency separately because it describes path diversity or retransmission above UDP, not additional independent events.

Sequence counters can wrap, and process restarts can reuse low values. The run ID prevents two generations from matching by accident. Encode integers with a documented byte order and message version. A corrupted identity should fail validation before it indexes a timestamp array, otherwise one malformed packet can pair unrelated times and create an extreme but believable sample.

Software clocks have different semantics

CLOCK_REALTIME follows civil system time and can step when synchronization software corrects it. CLOCK_MONOTONIC does not step with wall-clock changes but is adjusted in rate by time discipline. CLOCK_MONOTONIC_RAW exposes a hardware-based monotonic clock without NTP frequency adjustments. CLOCK_TAI represents International Atomic Time where the kernel's TAI offset is configured.

Two hosts cannot subtract CLOCK_MONOTONIC values because each starts from its own boot-related origin. CLOCK_MONOTONIC is excellent for durations within one host. Cross-host one-way subtraction typically uses synchronized CLOCK_REALTIME or CLOCK_TAI, or raw hardware PHC timestamps translated through measured offsets.

Leap seconds and time steps can corrupt a run. CLOCK_TAI avoids UTC leap discontinuity when the TAI offset is correct. Validate clock status before and after the interval and reject samples that cross an uncontrolled step.

Application timestamps include scheduling

Calling clock_gettime immediately before sendmsg measures when the sender thread reached that line, not when the packet left the NIC. The datagram can wait in a socket buffer, qdisc, driver queue, and hardware ring. The receiver's timestamp after recvmmsg includes NIC, stack, socket queue, wakeup, batching, and application scheduling.

That broad interval may match the operational question. A strategy consumes bytes in user space, so application receive time is relevant. The sender's application boundary may also represent the publication moment. The result should not be called wire latency.

Timestamp-call overhead and compiler placement matter at small scales. Benchmark clock_gettime on the target clock and host. vDSO calls are commonly fast, but page faults, migrations, and cache state affect tails. Keep the timestamp next to the measured operation and prevent avoidable logging or formatting in the hot loop.

Linux software receive timestamps reduce observer delay

SO_TIMESTAMPNS asks Linux to attach a software receive timestamp to datagrams as ancillary data returned by recvmsg. SO_TIMESTAMPING offers a more general set of software and hardware options. The kernel timestamp occurs before the application wakes, so comparing it with application dequeue time reveals socket residence and scheduling delay.

Use recvmsg or recvmmsg and parse control messages such as SCM_TIMESTAMPNS or SCM_TIMESTAMPING. Supply enough control-buffer space and count MSG_CTRUNC. A short ancillary buffer silently loses measurement data if the application ignores truncation.

Software receive timestamps still occur after NIC and some driver work. Their exact hook and clock follow kernel API semantics. They are more useful than a post-receive clock call for isolating host queue delay, but they are not physical ingress timestamps.

Hardware timestamps move toward the wire

NIC hardware timestamping uses a peripheral hardware clock, or PHC, and device support for selected packet classes. ethtool -T eth0 lists timestamping capabilities, the PTP hardware clock index, and supported transmit and receive filters. Not every virtual or physical NIC supports UDP event timestamping at the required layer.

SO_TIMESTAMPING selects software or hardware TX and RX reporting. Hardware transmit timestamps usually arrive asynchronously through the socket error queue, so the application must match them to packets. Receive timestamps arrive in ancillary data. SCM_TIMESTAMPING can contain software, transformed, and raw hardware times according to selected flags and kernel API.

Raw hardware timestamps live in the PHC clock domain. They cannot be subtracted from system time until the PHC-to-system offset is measured or the clock is disciplined appropriately. phc2sys and PTP system design determine that relation.

Clock synchronization needs an error budget

PTP can synchronize PHCs through hardware timestamps on a controlled Layer 2 or routed network, subject to profile and device support. NTP or NTS over general networks commonly provides lower precision but wider reach. GPS or another grandmaster source can anchor time while PTP distributes it locally.

Synchronization software reports an estimated offset, frequency correction, delay, and state. Those values are observations, not a hard bound. Asymmetric paths create offset error that a round-trip exchange cannot distinguish from clock difference. Hardware and topology calibration matter at microsecond and sub-microsecond targets.

Build a budget: sender timestamp uncertainty, sender clock offset, path asymmetry, receiver clock offset, receiver timestamp uncertainty, and timestamp-to-packet matching. Report the combined conservative bound. If the strategy needs 10 microsecond resolution and the clock budget is 30 microseconds, improve clocks before collecting more packets.

Round-trip time is not half one-way time

Ping and request-response probes measure round-trip time. Dividing by two assumes equal forward and reverse delay. Routing, queueing, serialization, firewalls, and link speeds can be asymmetric. The assumption may be approximately useful in a controlled symmetric fabric, but it is not a measurement of either direction.

UDP market data is one-way and may have no response packet. An ICMP echo follows different policies and packet sizes. A receiver acknowledgement adds application and scheduling delay to the reverse measurement. RTT remains valuable for reachability and change detection, not exact feed transit.

When synchronized clocks are unavailable, report RTT and local queue components separately. Do not relabel half-RTT as one-way latency. Honest uncertainty is more useful than false precision.

Captures need known timestamp sources

tcpdump and libpcap can use host software or adapter timestamp types depending on platform and device. tcpdump -J lists available timestamp types, and -j selects one. Hardware adapter timestamps may be synchronized or unsynchronized to system time. The label matters.

Packet captures can drop under load. tcpdump reports captured, received by filter, and dropped by kernel when it exits. Capture buffering and file writes add observer overhead. A passive hardware tap and capture appliance can provide a stronger wire reference, but it also needs calibrated time.

Use captures for packet identity and path comparison, while treating the timestamp source as a measurement instrument with a configuration and error. A pcap file is not automatically ground truth because its timestamps contain nine decimal places.

Quantiles need paired samples and loss reporting

Calculate p50, p90, p99, p99.9, maximum, and sample count from matched packets. Report missing sender IDs, receiver-only IDs, duplicates, reorder depth, and invalid timestamps. Excluding slow packets through an arbitrary timeout makes the tail look better. State the censoring rule.

Use an interval long enough to include bursts, idle transitions, route changes, and host housekeeping. Split results by time window and correlate with NIC, softnet, socket, and application queue counters. A bimodal distribution may map to CPU sleep or interrupt moderation rather than the network.

Histograms need sufficient resolution around the service objective without unbounded labels. Retain raw samples for a limited diagnostic window if policy permits. Aggregate latency by timestamp pair and run configuration, not by every slot as a permanent metric dimension.

Report confidence around rare quantiles. One million matched samples contain only one thousand observations above p99.9, and the maximum remains a single event. Repeat intervals across load conditions instead of treating one large aggregate as stationary. A route change or clock-state transition can make the combined histogram describe two different systems.

Calibration catches sign and scale errors

Put sender and receiver on the same host through loopback to validate packet identity and timestamp parsing. Then place them on one switch with known PTP status. Compare software and hardware timestamp pairs. Inject a known receiver sleep to verify that hardware-to-hardware time stays flat while hardware-to-application time grows.

Swap physical roles or cable paths to expose asymmetry. A consistent directional difference may be real link or device delay, but a clock-distribution asymmetry can produce the same symptom. Calibration equipment or a common-clock capture is required for a definitive separation.

Reject impossible data automatically. Negative wire times, abrupt offset jumps, missing TX timestamps, or a PHC state change should invalidate the affected interval. Silently clamping negative values to zero corrupts the distribution.

Production measurement should avoid the hot path

Embedding timestamps in every production packet changes the protocol unless the field already exists. Asking the sender for hardware TX timestamps on every packet can add driver and error-queue work. Capturing every packet to disk can create receive loss. Measurement overhead must be profiled.

Sample packets deterministically, such as one sequence in every 1,024, while keeping gap detection on all packets. Use preallocated rings and batch export. Keep timestamp correlation off the ingest core. Sampling reduces overhead but can miss microbursts, so pair it with queue and loss counters.

For shredstream.sh, no sender timestamp contract is defined in the product constants. A customer can measure arrival, host residence, decode time, and comparisons with another feed. Exact provider-to-customer one-way latency requires a shared timestamped test or synchronized observation at both endpoints.

Relative feed comparison still needs care. Match the same shred identity, take the earliest valid copy from each source according to a stated rule, and subtract timestamps on one receiver clock. This removes cross-host clock offset but measures path plus provider observation differences. It does not reveal either provider's absolute one-way transit time.