Solana shred feeds for indexers

6 stages produce a provisional index

An indexer turns ordered chain data into queryable records. A raw shred feed lets that work begin while a block is still being proposed. The result is a low-latency provisional index, not instant canonical truth.

shredstream.sh sends raw UDP from Frankfurt to one IP and port for $100 over 30 days. The feed measured 54.3 Mbps, 5,585 packets per second, and 1,216 mean packet bytes on 2026-08-31. At that rate, one destination receives 17.6 TB in 30 days. Retaining decoded rows, raw payloads, indexes, and replicas can require much more storage.

Indexing latency differs from trading latency. Detection, decode, and local submission can finish in milliseconds. Canonical commitment takes chain time. The table keeps both scales visible so an API cannot label proposed data as final merely because it arrived early.

Early rows need an explicit state

Solana has no public mempool. Transactions in shreds are part of a leader's proposed block. They can fail, live on a fork that disappears, or belong to a skipped slot. An indexer should expose this status rather than flattening every observation into one permanent table.

A useful record lifecycle is observed, decoded, processed, confirmed, finalized, or orphaned. Exact names can differ, but the transition must be monotonic within one fork and reversible when a provisional fork loses. Consumer queries should choose a minimum commitment level.

The raw observation timestamp answers when the receiver saw bytes. Decode timestamp answers when a transaction became available. Database commit timestamp answers when a query could return it. Chain commitment answers whether the network accepted it at a selected level. Those are four different clocks.

Fields preserve provenance

Slot and shred index locate fragments. FEC set index connects data and coding shreds for recovery. The variant records whether a stored payload came directly from data or was reconstructed. Completion flags distinguish usable entry boundaries from the final data shred for a slot.

Leader signature and shred version support authentication and format routing. Transaction signature becomes a stable query key, but it needs slot and fork context while provisional. Account keys support address history and change indexes. Program IDs and instruction discriminators support semantic tables for known protocols.

Preserve raw instruction bytes and parser version beside decoded columns. A parser upgrade can reinterpret old data. Without provenance, correcting a semantic table becomes guesswork. Arrival, recovery, decode, and persistence timestamps make freshness measurable.

An indexer should also record gaps. Absence of a transaction from an incomplete slot is not evidence that it did not exist. Completeness metadata belongs beside data access, not only in an internal dashboard.

A defensible latency budget

The first five stages total 1.8 to 16.2 milliseconds. A decoded row can therefore be queryable within tens of milliseconds of its required bytes arriving under the stated assumptions. The 0.4 to 13 second land range is intentionally different in scale. It is a planning interval for processed through stronger commitment, not a product guarantee or a fixed protocol constant.

Detection performs header classification and cheap validation. It should not allocate a full object graph per packet. Decode includes grouping, reorder, optional erasure recovery, entry deserialization, transaction parsing, and supported semantic extraction. Publish separate distributions for raw transaction availability and enriched rows.

Decision is a data-quality branch. Known valid records go to provisional storage. Duplicates increment counters without duplicate rows. Unknown versions or malformed content go to quarantine. Incomplete FEC sets stay pending until a deadline, then become explicit gaps.

Signing is zero for the common case because an indexer is not sending an on-chain transaction. If the service signs batch manifests for tamper evidence, signing should occur after batching and must not stall receive. Calling this stage zero is more honest than inventing cryptographic work that the product does not need.

Submission means durable local ingestion, not chain submission. A memory queue can acknowledge in fractions of a millisecond but loses data on process failure. A replicated log or synchronous database commit costs more. State the durability boundary beside the latency number.

Land means the indexed transaction reaches the selected chain commitment. The indexer observes rather than causes that transition. Commitment can be delayed or never occur for a provisional fork.

Work one slot through the lifecycle

Assume the receiver gets the first useful shred for slot 410,000,100 at time zero. It classifies the header in 0.4 milliseconds. The entry containing a target swap spans two available data shreds and becomes decodable 1.7 milliseconds later. Parsing and account expansion take another 1.1 milliseconds.

The provisional record is assigned slot, transaction order, signature, accounts, program, raw instruction bytes, parser version, and receive timestamps. A 200-row batch appends to the local log in 2.5 milliseconds. The API can expose the row at observed commitment roughly 5.7 milliseconds after all required bytes became available.

Later chain observations show the slot processed at 430 milliseconds, confirmed at 1.8 seconds, and finalized at 12.4 seconds in this example. The indexer updates commitment without rewriting the raw provenance. If the slot had lost its fork, the record would move to orphaned and disappear from queries requiring processed or above.

The example separates data readiness from chain status. A single field named indexed_at cannot answer both questions.

Storage arithmetic starts at 17.6 TB

The published 17.6 TB per 30 days is the measured packet-byte stream at the product rate. Raw archival framing, timestamps, file headers, checksums, compression metadata, and replication add capacity. Semantic database rows and secondary indexes add a separate footprint.

An indexer does not need to retain every raw packet forever. A circular capture of recent hours can support decoder debugging while long-term storage keeps ordered transaction bytes and provenance. The retention decision should follow recovery requirements and cost, not a belief that raw data compresses well. Signatures and encoded payloads can limit generic compression gains.

At 5,585 packets per second, one 32-byte per-packet metadata record adds about 14.3 GiB per day before database overhead. Per-packet logging is both a storage cost and a latency risk. Aggregate metrics and sample diagnostics.

UDP changes correctness duties

UDP provides no ordering, retransmission, or backpressure. The receiver must detect duplicates and missing indexes. Coding shreds can reconstruct absent data when enough recovery symbols arrive. A repair protocol or secondary source is outside this delivery contract.

Kernel buffers absorb finite stalls. Application queues absorb downstream variance only while receive continues. Neither fixes sustained throughput below the feed rate. Track interface drops, kernel UDP errors, socket overflow counters, application drops, FEC completion, oldest queued age, and database lag.

Bound each stage. When semantic enrichment falls behind, preserve raw ordered transactions and defer enrichment instead of blocking receive. When persistence falls behind, the system needs an explicit shed or failover policy. UDP will not ask the sender to wait.

Traffic comes from 64.130.40.90. Allow inbound UDP from that address to the chosen port. The activation challenge expires after 600 seconds. It proves current reachability for a datagram, not loss-free ingestion or durable indexing.

Parser versions are data

Program layouts and Solana shred formats evolve. Route unknown shred versions and instruction variants to quarantine. Store decoder build and schema version with output. Reprocess retained raw bytes after a parser correction and compare row counts and field changes.

Avoid destructive in-place reinterpretation without an audit trail. A derived table can be rebuilt. The original transaction bytes, slot context, and parser version should remain available for disputed results. Deterministic replay is the indexer's strongest correctness test.

Use fixtures for clean, reordered, duplicated, recovered, and incomplete FEC sets. Test forks and skipped slots. Compare transaction signatures and ordering against a canonical source. Measure every mismatch instead of assuming an early feed is correct because it is fast.

Serving APIs without overstating certainty

Every response should expose commitment or require it as a query parameter. Streaming clients need retraction or status-update events for provisional data. Cached responses must include commitment in the key. Otherwise an observed row can remain in a cache after its fork disappears.

Freshness service levels should name the start and end points. Packet arrival to provisional API is controllable by the indexer. Leader production to packet arrival is not fully observable without comparable remote timestamps. Provisional API to finalized status is a chain process, not an ingest service promise.

Rate limits and slow consumers must not feed back into packet ingestion. Serve from materialized stores or independent queues. One client requesting a broad address history should never delay shred receive.

When raw shreds are not the right source

An indexer serving finalized historical queries gains little from millisecond proposed data. A block or transaction source with replay can reduce operational complexity and provide stronger completeness. Raw shreds fit products that can use provisional state and maintain a reconciliation pipeline.

The service is Frankfurt-only and delivers to one IP and port per destination. It provides no retransmission, ordering, backpressure, historical replay, or regional redundancy. An archival indexer needs additional sources and storage policies if gaps are unacceptable.

Acceptance is a replay and restart test

Capture a bounded interval and run it through the decoder twice. The resulting ordered transaction identities, parser versions, gap records, and provisional status transitions should match. Then reorder datagrams, insert duplicates, remove recoverable data shreds, and confirm that the canonical decoded output remains the same while recovery metadata changes.

Restart every process at a different boundary. Restart after packet receipt but before decode, after decode but before durable append, and after provisional publication but before commitment. The index should avoid duplicate rows and should never lose the ability to retract a provisional fork. Document which acknowledgement survives each crash.

Compare supported canonical transactions with an independent later source. A mismatch needs one of four labels: source gap, decoder error, parser scope, or commitment-policy difference. An unlabeled mismatch is not an acceptable residual. Track counts over time and gate deployments when they regress.

Load testing should exceed the measured 5,585 packets per second in short bursts and should stall persistence for the declared buffer interval. The receiver must expose whether it shed enrichment, raw input, or client delivery. A queue that grows without a time limit passes throughput briefly by accumulating latency.

The honest outcome is a two-speed index. Milliseconds make provisional records queryable. Seconds advance them toward canonical commitment. Clear provenance lets consumers choose between freshness and certainty without being misled about either.