Solana shred feeds for research desks

6 stages preserve an observation

A research desk needs primary data, documented measurement boundaries, and reproducible transformations. Raw shreds offer a view of Solana block propagation and proposed execution before normal block interfaces. They also arrive over an unreliable datagram transport and may describe forks that never become canonical.

shredstream.sh delivers raw UDP from Frankfurt to one IP and port for $100 over 30 days. The feed measured 54.3 Mbps, 5,585 packets per second, and 1,216 mean packet bytes on 2026-08-31. That rate produces 17.6 TB over 30 days before capture framing, indexes, replicas, and derived datasets.

For research, a millisecond is useful only when its clock, host, receive layer, packet identity, and missing-data policy are known. A smaller undocumented timestamp can be less informative than a later reproducible record.

Shreds are early proposed data

Solana has no public mempool. Transactions observed in shreds are already part of a leader's proposed block. They can provide an early execution sequence, but they do not reveal all private submissions, dropped transactions that never reached the leader, or intent before inclusion.

Slots can be skipped. Forks can lose. Transactions can fail. A study comparing shreds with confirmed blocks must retain observed-only events rather than deleting them as mistakes. Those differences may be the object of the research.

The feed reflects one relay origin in Frankfurt and one receiver path. It is not a neutral view from every Turbine layer or geography. Propagation conclusions must name that vantage point. Comparing it with another host adds a second path, not a view from nowhere.

Fields establish reproducibility

Raw packet bytes and the earliest available receive timestamp are the primary observation. Slot, shred index, FEC set index, and variant make the packet placeable. Leader signature authenticates the proposed producer. Shred version segments formats across network changes.

Completion flags show entry and slot boundaries. Recovery status should record whether data arrived directly or was reconstructed from coding shreds. A derived transaction without that distinction can bias packet-loss research.

Transaction signature joins the proposal to later canonical history. Account keys and program IDs support selected populations. Raw instruction data permits re-parsing when program semantics change. Parsed columns should include decoder and schema versions.

Host identity, kernel version, NIC, timestamp mode, clock source, measured clock offset, capture filter, and software build are research fields even though they are not inside a shred. Without them, latency results cannot be compared across runs.

A defensible latency budget

The first five stages total 2.7 to 41.5 milliseconds. Research can allow more decode and persistence time than a trading strategy while keeping receive isolated. The land interval is intentionally wider and measured in chain time. It joins an observation to canonical outcome rather than causing execution.

Detection records and classifies. Decode reconstructs structured content. Decision applies retention and data-quality policy. Signing is absent unless the desk wants tamper-evident batch manifests. Submission means durable research storage, not transaction submission. Land means later chain status.

These definitions make the same six-stage frame useful without suggesting that a research process sends trades. Zero is a valid signing measurement when no signature is required.

Work a 60-second capture example

At the measured mean, 60 seconds contains about 335,100 packets because 5,585 multiplied by 60 equals 335,100. Packet payload bytes total about 407,481,600 because 335,100 multiplied by 1,216 equals that value.

Suppose the receiver records 334,950 datagrams from 64.130.40.90 during the exact interval. The difference from the mean-rate expectation is 150 packets, but it is not valid to label those 150 as loss. The feed rate varies. Loss requires expected sequence analysis, FEC-set completeness, sender or independent-path evidence, and receiver drop counters.

The decoder finds 120 duplicate datagrams, 18 missing data shred indexes later recovered through coding shreds, and two incomplete FEC sets. The dataset should report all four quantities: received datagrams, unique shred identities, recovered data, and unrecovered gaps. A clean transaction table alone erases the transport result.

Later reconciliation marks 98.7 percent of decoded transactions on the surviving chain and the rest on losing or incomplete branches in this illustrative sample. The exact percentage is not a product fact. The method and category definitions are the useful parts.

Storage has more than one byte count

The 17.6 TB figure is the measured 30-day feed calculation. A packet capture adds record headers and perhaps interface metadata. Per-packet timestamps add storage at a rate multiplied by more than 14 billion monthly packets. Filesystem allocation, checksums, compression framing, and replication add more.

Derived entry and transaction tables can duplicate raw payload information. Secondary indexes can exceed base tables for account-rich workloads. A two-copy archive starts above 35.2 TB of packet bytes before overhead. Capacity plans should list raw, metadata, derived, index, replica, and free-space reserve separately.

Retention can be tiered. Keep full packet captures for a shorter reproducibility window, ordered reconstructed bytes for longer, and compact canonical facts for the research horizon. Deletion policy should preserve the source needed to reproduce published claims.

Compression needs measurement. Signatures and encoded data contain high entropy. A compressor that saves little while consuming a receive core can worsen data quality. Compress after durable buffering or on separate workers.

Packet loss is a result, not an inconvenience

UDP provides no ordering, retransmission, or backpressure. Missing and duplicated packets must remain visible in the dataset. Coding shreds can recover data, but recovery changes the observation category. An entry that became readable after recovery has a different readiness time from one received cleanly.

Track NIC hardware drops, interface drops, kernel UDP errors, socket overflow, application queue drops, malformed headers, signature failures, missing indexes, late arrivals, recovered sets, and incomplete sets. Counters live at different layers and will not always match.

Bound queues. If storage stalls, preserve receive by switching to an explicit sampling or drop policy and record the interval. Silent overload creates selection bias. A capture that misses the busiest slots can support the opposite conclusion from reality.

The source IP is 64.130.40.90. The firewall should allow inbound UDP from that address to the selected port. The destination challenge is valid for 600 seconds and tests immediate reachability. It does not establish sample completeness.

Timing claims need comparable boundaries

Kernel receive timestamps reduce scheduling noise relative to user-space timestamps. Hardware NIC timestamps can move the boundary closer to the wire when supported and configured correctly. A decoder completion timestamp measures software work, not network arrival.

Use monotonic clocks for intervals. Use synchronized wall clocks for comparing hosts, and retain clock offset plus uncertainty. NTP can support many millisecond studies. Sub-millisecond one-way claims may require PTP, hardware timestamping, and a carefully calibrated path.

Leader emission time is not normally known from the receiver's local clock alone. Slot-relative estimates introduce assumptions. State those assumptions and avoid presenting propagation time as directly measured when only arrival differences were observed.

Packet capture itself can perturb the system through copies and disk traffic. Run controlled tests with capture enabled and disabled. Record CPU, drops, and queue age. Research instrumentation belongs inside the performance model.

Sampling can preserve the wrong thing

Uniform packet sampling reduces storage but can break entry reconstruction because transactions span shreds. Sampling complete slots, complete FEC sets, or bounded time windows often preserves more analyzable structure. The correct unit depends on the question.

A study of packet sizes can sample datagrams. A study of transaction ordering needs complete ordered data. A study of recovery needs both data and coding shreds plus gap provenance. A study of market reaction needs synchronized external sources and explicit asset selection.

Publish inclusion and exclusion rules before looking at the result where practical. Retain counts for filtered populations. An account allowlist makes the dataset smaller but prevents network-wide conclusions.

Reconciliation prevents survivor bias

Join every observed transaction signature and slot with a later canonical source. Preserve observed-only, failed, orphaned, and canonical categories. Do not discard losing forks from a propagation study. Do not count them as settled volume in an economics study.

Parser failures should remain a category. Re-run raw data after decoder changes and version outputs. A result that depends on one parser build should name it. Store checksums for source files and transformation manifests for derived tables.

An optional signed manifest can bind file names, hashes, capture interval, host, filter, and software versions. Signing the manifest does not prove the capture was complete. It helps show that the archived bytes have not changed since the manifest was made.

Ethical and legal scope remains real

Raw chain data can still become sensitive when combined with identity labels, exchange records, or unpublished datasets. Research protocols should consider privacy, retention, access, and responsible disclosure. Public address activity does not make every off-chain attribution accurate.

Studies of MEV should distinguish mechanisms and externalities. Sandwiching worsens the surrounded user's execution and transfers value through ordering. This page does not provide a recipe for harmful strategies. Measurement should not be framed as endorsement.

Document funding, conflicts, missing data, and failed hypotheses. Millisecond precision cannot compensate for biased sampling or unstated definitions.

When raw shreds are not the right dataset

Questions about finalized balances, long-term protocol revenue, or yearly user cohorts rarely require packet-level input. An archival block source with stable schemas can reduce cost and provide replay. Raw shreds fit questions about propagation, earliest observable state, loss, forks, and pipeline timing.

Frankfurt-only delivery and one IP and port define one vantage point. There is no retransmission, ordering, backpressure, historical replay, or multi-region capture in the plan. A comparative network study needs additional vantage points under the researcher's control.

Publication needs an artifact trail

A published result should identify capture interval, receiver region, filter, clock method, software versions, missing-data rule, parser coverage, commitment filter, and transformation code. Report packet and transaction counts at every exclusion step. Provide hashes for retained source artifacts when sharing the bytes is not possible.

Recompute tables from a clean environment and compare them byte for byte or within stated floating-point tolerances. Ask a second analyst to reproduce one central figure from the method and intermediate counts. Disagreement is evidence about the pipeline, not a formatting problem.

Freeze the dataset used for a paper or note. Continuing live corrections can make a published number irreproducible. New parser findings should create a versioned correction with the effect quantified.

The defensible output is not a claim that raw equals truth. It is a dataset that preserves what one receiver observed, when it observed it, what was missing, how it decoded the bytes, and what the chain later accepted.