Benchmark Your Shred Provider Yourself

Every benchmark in this market is marketing

Search for Solana shred latency benchmarks and you will find several. They are competently produced, they contain real measurements, and every one of them was published by a company that came first in it. That is not an accusation of fraud. It is a structural observation: a vendor chooses the regions, the window, the comparison set, the metric, and when to publish. Each of those choices has a defensible justification and each of them moves the result. Nobody has to lie for the output to be useless to you. There is also a harder problem underneath. A shred feed's value is where it lands, and it lands at your destination, not the vendor's. A benchmark run from someone else's host in someone else's rack tells you about their network path. So the answer is not a better vendor benchmark. It is measuring it yourself, which is cheaper than it sounds.

The clock problem, and why it disappears

The obvious way to benchmark two feeds is to run each on the machine nearest it and compare timestamps. This does not work. You are now comparing two clocks, and NTP on ordinary hardware drifts by more than the differences you are trying to detect. At the millisecond scale, clock skew is not a correction you apply - it is the entire signal. Hardware timestamping and PTP solve this properly, and if you already have that infrastructure, use it. If you do not, use one host and two ports. One CLOCK_MONOTONIC reading can time every packet from both feeds, so the comparison has no second-clock error. The cost of this design is that you measure delivery to one location. That is a feature. You do not care about a feed averaged across a global footprint; you care about the feed arriving at the box that executes.

The metrics, in the order you should read them

Most people jump to p50 latency. It is the third most important number here.

1. Coverage - did you get all the shreds?

Coverage is the fraction of observed shred keys delivered by a feed. Reject an apparently fast source if its set is incomplete.

2. Kernel drops - are you measuring the network or your own host?

If your socket is dropping packets, the benchmark is measuring receiver failure. shredbench reads the per-socket counter in /proc/net/udp; a non-zero value means the host must be fixed before another run.

3. Win rate - how often did this feed arrive first?

For every shred in the union, which feed delivered it first. This is closer to trading reality than a median. A feed may lose on p50 yet win more individual races because first arrival is not an average.

4. Delta percentiles - p50 and, more importantly, p99

The median tells you about a typical shred. p99 exposes congestion, where feeds diverge most. A feed with a calm median and a long tail fails during the periods that matter.

The harness

git clone https://github.com/YOURORG/shredbenchcd shredbench./shredbench.py --feed provider-a=20000 --feed provider-b=20001                 --baseline provider-a --seconds 3600                 --json run-$(date +%F).json

Output:

window 3600.0s   union 2841503 shreds   baseline 'provider-a'

arrival delta vs 'provider-a' (ms, negative = faster than baseline)

(Shape of the output, not a published result. The point of the tool is that you generate your own.) It has no dependencies beyond Python 3.8+, takes any number of feeds, and writes JSON so you can run it from cron and keep a history.

How it identifies shreds

Only the common header, per solana-foundation/specs. No ledger dependency, decoding, or FEC reconstruction:

(slot, index) is the join key, and it is stable across providers - the same shred forwarded by two vendors carries the same pair, which is what makes the comparison possible at all. The variant byte separates data shreds (0x8_ merkle, 0xa5 legacy) from coding shreds (0x4_ merkle, 0x5a legacy), so you can confirm both feeds are sending you the same mix rather than one quietly withholding parity.

More on the wire format in our shred format reference.

Testing it before you pay for anything

The repo ships fakefeed.py, a synthetic generator with configurable lag and loss, so you can confirm the harness behaves before pointing money at it:

# terminal 1 - feed b lags 0.6 ms and drops 3%./fakefeed.py --to fast=20010:0:0.0 --to slow=20011:0.6:0.03 --slots 500 # terminal 2./shredbench.py --feed a=20010 --feed b=20011 --seconds 30 --bind 127.0.0.1

Feed b should come back around 0.6 ms behind at roughly 97% coverage. If it does not, the harness is wrong and you should open an issue.

Running it so the answer means something

Raise the receive buffer first. Linux caps SO_RCVBUF at the host's configured ceiling and reports an adjusted accounting value, which confuses first-time measurements. shredbench prints the effective size.

sudo sysctl -w net.core.rmem_max=134217728

Use a representative window

A quiet sample can hide queue pressure. Include the trading conditions that matter to your strategy and rerun the comparison on several dates.

Run more than once

One window is an anecdote. Three windows across different days is a finding.

Do not change anything else

Same host, same NIC, same kernel, same time period. If you compare a feed you measured last month against one you measured today, you have measured the month.

Check that both feeds are actually live

A provider whose feed died at hour two will post excellent latency on the shreds it did send.

What this cannot tell you

The honest limits, because a measurement tool that oversells itself is the thing we are complaining about.

It measures arrival, not usefulness

Shreds arriving first still have to be reconstructed and decoded before you can act. If one provider ships raw UDP and another ships decoded transactions, this compares the packets, not the end-to-end time to a tradeable signal. For that you need to instrument your own pipeline.

It measures one destination

That is the design, but it means the result does not generalise. Your Frankfurt result says nothing about Tokyo.

It cannot see loss upstream of both feeds

If every source misses the same shred, the union cannot reveal it. Use confirmed RPC data as an occasional external correctness check.

It says nothing about reliability over time

A day of measurement tells you about a day. Uptime is a different question and a longer one.

We wrote it

We tried to make that not matter - the methodology is written down, the code is short enough to read in ten minutes, and it does not know which feed is ours. But you should read it rather than trust it, and that is the correct response to a benchmark published by a vendor, including this one.

An invitation

If you sell shred delivery and find a methodological error, send a reproducible case and we will review it publicly. If you think it measures you fairly, run it and publish the numbers. A market where three providers publish comparable results from the same tool is better for buyers than one where each publishes a benchmark they won, and we will happily publish results where we lose.