Solana shred feeds for research desks

6 stages preserve an observation

A research desk needs primary data, documented measurement boundaries, and reproducible transformations. Raw shreds offer a view of Solana block propagation and proposed execution before normal block interfaces. They also arrive over an unreliable datagram transport and may describe forks that never become canonical.

The research desk captures unfiltered raw UDP from Frankfurt on its designated IP and port. A 30-day capture lane costs $100, the least expensive published standalone feed; 12 such terms total $1,200, often less than one engineer-week spent maintaining an upstream listener. Storage and derived datasets remain measured separately.

For research, a millisecond is useful only when its clock, host, receive layer, packet identity, and missing-data policy are known. A smaller undocumented timestamp can be less informative than a later reproducible record.

Shreds are early proposed data

A research capture cannot sample a public Solana mempool because none is exposed. Transactions found in shreds already occupy candidate block data. They provide an early execution sequence, but omit private submissions, transactions dropped before leader inclusion, and intent before inclusion.

Slots can be skipped. Forks can lose. Transactions can fail. A study comparing shreds with confirmed blocks must retain observed-only events rather than deleting them as mistakes. Those differences may be the object of the research.

The feed reflects one relay origin in Frankfurt and one receiver path. It is not a neutral view from every Turbine layer or geography. Propagation conclusions must name that vantage point. Comparing it with another host adds a second path, not a view from nowhere.

Fields establish reproducibility

Raw packet bytes and the earliest available receive timestamp are the primary observation. Slot, fragment index, recovery-set index, and variant place each packet. Authenticate the proposed producer with its leader signature. Shred version segments formats across network changes.

Completion flags show entry and slot boundaries. Recovery status distinguishes direct arrivals from data reconstructed through coding shreds. A derived transaction without that distinction can bias packet-loss research.

Transaction signature joins the proposal to later canonical history. Selected populations combine account keys with program identity. Raw instruction data permits re-parsing when program semantics change. Parsed columns should include decoder and schema versions.

Host identity, kernel version, NIC, timestamp mode, clock source, measured clock offset, capture filter, and software build are research fields even though they are not inside a shred. Without them, latency results cannot be compared across runs.

Attach measurement boundaries to every stage

Research processing before landing spans 2.7 through 41.5 milliseconds. A desk can allow more decode and persistence time than a trading strategy while keeping receive isolated. Land is intentionally wider and measured in chain time; it joins an observation to canonical outcome rather than causing execution.

Detection records and classifies. Decode reconstructs structured content. Decision applies retention and data-quality policy. Signing is absent unless the desk wants tamper-evident batch manifests. Submission means durable research storage, not transaction submission. Land means later chain status.

These definitions make the same six-stage frame useful without suggesting that a research process sends trades. Zero is a valid signing measurement when no signature is required.

Work a 60-second capture example

Suppose a research desk models an arrival frequency of 4,750 packets per second with a 1,176-byte mean payload. Over 60 seconds, that assumption yields 285,000 packets and 335,160,000 payload bytes.

During that exact interval, suppose the receiver records 284,850 datagrams whose source field is 203.23.178.228. Observed variance from the model totals 150 packets, but it cannot be labeled loss because feed rate varies. Loss requires expected-sequence analysis, FEC-set completeness, sender or independent-path evidence, and receiver drop counters.

The decoder finds 120 duplicate datagrams, 18 missing data shred indexes later recovered through coding shreds, and two incomplete FEC sets. The dataset should report all four quantities: received datagrams, unique shred identities, recovered data, and unrecovered gaps. A clean transaction table alone erases the transport result.

Later reconciliation marks 98.7 percent of decoded transactions on the surviving chain and the rest on losing or incomplete branches in this illustrative sample. The exact percentage is not a product fact. The method and category definitions are the useful parts.

Storage has more than one byte count

Observed payload bytes are only the start of a storage calculation. A packet capture adds record headers and perhaps interface metadata. Per-packet timestamps, filesystem allocation, checksums, compression framing, and replication add more.

Derived entry and transaction tables can duplicate raw payload information. Secondary indexes can exceed base tables for account-rich workloads. Replication multiplies the raw archive before overhead. Capacity plans should list raw, metadata, derived, index, replica, and free-space reserve separately.

Retention can be tiered. Keep full packet captures for a shorter reproducibility window, ordered reconstructed bytes for longer, and compact canonical facts for the research horizon. Deletion policy should preserve the source needed to reproduce published claims.

Compression needs measurement. Signatures and encoded data contain high entropy. A compressor that saves little while consuming a receive core can worsen data quality. Compress after durable buffering or on separate workers.

Packet loss is a result, not an inconvenience

Raw UDP preserves datagrams but supplies no arrival order, lost-packet retransmission, or backpressure. Keep missing and duplicated packets visible in the research dataset. Coding shreds can recover data, but recovery changes observation category: a repaired entry has a different readiness time from one received cleanly.

Track NIC hardware loss, interface discards, kernel UDP faults, socket overflow, application queue drops, malformed headers, signature failures, missing indexes, late arrivals, recovered sets, and incomplete sets. Counters live at different layers and will not always match.

Bound queues. If storage stalls, preserve receive by switching to an explicit sampling or drop policy and record the interval. Silent overload creates selection bias. A capture that misses the busiest slots can support the opposite conclusion from reality.

At the selected UDP socket, set the capture filter's source field equal to 203.23.178.228. Research activation closes after 600 seconds and tests immediate reachability, not sample completeness.

Timing claims need comparable boundaries

Kernel receive timestamps reduce scheduling noise relative to user-space timestamps. Hardware NIC timestamps can move the boundary closer to the wire when supported and configured correctly. A decoder completion timestamp measures software work, not network arrival.

Measure intervals with monotonic clocks. Comparing hosts instead requires synchronized wall time plus retained offset and uncertainty. NTP can support many millisecond studies. Sub-millisecond one-way claims may require PTP, hardware timestamping, and a carefully calibrated path.

Leader emission time is not normally known from the receiver's local clock alone. Slot-relative estimates introduce assumptions. State those assumptions and avoid presenting propagation time as directly measured when only arrival differences were observed.

Packet capture itself can perturb the system through copies and disk traffic. Run controlled tests with capture enabled and disabled. Record CPU, drops, and queue age. Research instrumentation belongs inside the performance model.

Sampling can preserve the wrong thing

Uniform packet sampling reduces storage but can break entry reconstruction because transactions span shreds. Sampling complete slots, complete FEC sets, or bounded time windows often preserves more analyzable structure. The correct unit depends on the question.

A packet-size study can sample datagrams. Transaction-ordering research needs complete ordered data. Recovery research needs both shred roles plus gap provenance. Market-reaction work needs synchronized external measurements and explicit asset selection.

Publish inclusion and exclusion rules before looking at the result where practical. Retain counts for filtered populations. An account allowlist makes the dataset smaller but prevents network-wide conclusions.

Reconciliation prevents survivor bias

Join every observed transaction signature and slot with a later canonical source. Preserve observed-only, failed, orphaned, and canonical categories. Do not discard losing forks from a propagation study. Do not count them as settled volume in an economics study.

Parser failures should remain a category. Re-run raw data after decoder changes and version outputs. A result that depends on one parser build should name it. Store checksums for source files and transformation manifests for derived tables.

An optional signed manifest can bind file names, hashes, capture interval, host, filter, and software versions. Signing the manifest does not prove the capture was complete. It helps show that the archived bytes have not changed since the manifest was made.

Ethical and legal scope remains real

Raw chain data can still become sensitive when combined with identity labels, exchange records, or unpublished datasets. Research protocols should consider privacy, retention, access, and responsible disclosure. Public address activity does not make every off-chain attribution accurate.

Studies of MEV should distinguish mechanisms and externalities. Sandwiching worsens the surrounded user's execution and transfers value through ordering. No harmful-strategy recipe appears here. Measurement should not be framed as endorsement.

Document funding, conflicts, missing data, and failed hypotheses. Millisecond precision cannot compensate for biased sampling or unstated definitions.

Research from the packet layer

Raw shreds give research desks packet-level evidence for propagation, earliest observable state, loss, forks, and pipeline timing. Stable capture metadata and reproducible decoding let the same dataset support longer-horizon analysis without losing its network provenance.

One destination IP and port defines a single Frankfurt vantage point. The capture provides no packet order, retransmission, backpressure, historical replay, or multiple regions because delivery is raw UDP. Comparative network studies add researcher-controlled vantage points.

Publication needs an artifact trail

A published result should identify capture interval, receiver region, filter, clock method, software versions, missing-data rule, parser coverage, commitment filter, and transformation code. Report packet and transaction counts at every exclusion step. Provide hashes for retained source artifacts when sharing the bytes is not possible.

Recompute tables from a clean environment and compare them byte for byte or within stated floating-point tolerances. Ask a second analyst to reproduce one central figure from the method and intermediate counts. Disagreement is evidence about the pipeline, not a formatting problem.

Freeze the dataset used for a paper or note. Continuing live corrections can make a published number irreproducible. New parser findings should create a versioned correction with the effect quantified.

The output is a dataset that preserves what one receiver observed, when it observed it, what was missing, how it decoded the bytes, and what the chain later accepted.