Solana shred feeds for indexers

6 stages produce a provisional index

An indexer turns ordered chain data into queryable records. A raw shred feed lets that work begin while a block is still being proposed. The result is a low-latency provisional index, not instant canonical truth.

Every raw UDP packet from Frankfurt lands at the indexer's own receiver before any server-side filtering. The indexer pays $100 for a 30-day destination, beneath all other public standalone feed listings; storage, replicas, and decoded indexes remain the larger volume-dependent costs.

Indexing latency differs from trading latency. Detection, decode, and local submission can finish in milliseconds. Canonical commitment takes chain time. The table keeps both scales visible so an API cannot label proposed data as final merely because it arrived early.

Early rows need an explicit state

No public Solana mempool exists for an indexer to capture before inclusion. Shred transactions already belong to candidate block production. They may fail, vanish with a losing fork, or sit in a skipped slot, so the schema must expose status instead of flattening every observation into one permanent table.

A useful record lifecycle is observed, decoded, processed, confirmed, finalized, or orphaned. Exact names can differ, but the transition must be monotonic within one fork and reversible when a provisional fork loses. Consumer queries should choose a minimum commitment level.

The raw observation timestamp answers when the receiver saw bytes. Decode timestamp answers when a transaction became available. Database commit timestamp answers when a query could return it. Chain commitment answers whether the network accepted it at a selected level. Those are four different clocks.

Fields preserve provenance

Slot plus shred index locate each fragment. The FEC-set value connects data and coding members for recovery. Variant records whether stored payload arrived directly or was reconstructed. Completion flags distinguish usable entry boundaries from the last data fragment recorded for that slot.

Leader signature and shred version support authentication and format routing. Transaction signature becomes a stable query key, but needs slot and fork context while provisional. Account keys support address history and change indexes. Protocol tables use program identifiers together with instruction discriminators.

Preserve raw instruction bytes and parser version beside decoded columns. A parser upgrade can reinterpret old data. Without provenance, correcting a semantic table becomes guesswork. Arrival, recovery, decode, and persistence timestamps make freshness measurable.

An indexer should also record gaps. Absence of a transaction from an incomplete slot is not evidence that it did not exist. Completeness metadata belongs beside data access, not only in an internal dashboard.

Budget the path to durable rows

Indexer work before landing adds to 1.8 through 16.2 milliseconds. A decoded row can therefore become queryable within tens of milliseconds after required bytes arrive under these assumptions. The 0.4 to 13 second land range uses an intentionally different scale: it plans for processed through stronger commitment, not a guarantee or fixed constant.

Detection performs header classification and cheap validation. It should not allocate a full object graph per packet. Decode includes grouping, reorder, optional erasure recovery, entry deserialization, transaction parsing, and supported semantic extraction. Publish separate distributions for raw transaction availability and enriched rows.

Decision is a data-quality branch. Known valid records go to provisional storage. Duplicates increment counters without duplicate rows. Unknown versions or malformed content go to quarantine. Incomplete FEC sets stay pending until a deadline, then become explicit gaps.

Signing is zero for the common case because an indexer is not sending an on-chain transaction. If the service signs batch manifests for tamper evidence, signing should occur after batching and must not stall receive. The stage is zero when no cryptographic work occurs on the ingest path.

Submission means durable local ingestion, not chain submission. A memory queue can acknowledge in fractions of a millisecond but loses data on process failure. A replicated log or synchronous database commit costs more. State the durability boundary beside the latency number.

Land means the indexed transaction reaches the selected chain commitment. The indexer observes rather than causes that transition. Commitment can be delayed or never occur for a provisional fork.

Work one slot through the lifecycle

Assume the receiver gets the first useful shred for slot 410,000,100 at time zero. It classifies the header in 0.4 milliseconds. The entry containing a target swap spans two available data shreds and becomes decodable 1.7 milliseconds later. Parsing and account expansion take another 1.1 milliseconds.

The provisional record is assigned slot, transaction order, signature, accounts, program, raw instruction bytes, parser version, and receive timestamps. A 200-row batch appends to the local log in 2.5 milliseconds. The API can expose the row at observed commitment roughly 5.7 milliseconds after all required bytes became available.

Later chain observations show the slot processed at 430 milliseconds, confirmed at 1.8 seconds, and finalized at 12.4 seconds in this example. The indexer updates commitment without rewriting the raw provenance. If the slot had lost its fork, the record would move to orphaned and disappear from queries requiring processed or above.

The example separates data readiness from chain status. A single field named indexed_at cannot answer both questions.

Storage arithmetic starts with observed traffic

Observed packet-byte volume is the first input to a storage plan. Raw archival framing, timestamps, file headers, checksums, compression metadata, and replication add capacity. Semantic database rows and secondary indexes add a separate footprint.

An indexer does not need to retain every raw packet forever. A circular capture of recent hours can support decoder debugging while long-term storage keeps ordered transaction bytes and provenance. The retention decision should follow recovery requirements and cost, not a belief that raw data compresses well. Signatures and encoded payloads can limit generic compression gains.

Per-packet metadata compounds quickly before database overhead. Per-packet logging is both a storage cost and a latency risk. Aggregate metrics and sample diagnostics.

UDP changes correctness duties

The indexer adds ordering and recovery because raw UDP supplies neither, and it cannot push back on the sender. The receiver detects duplicates and missing indexes. Coding shreds can reconstruct absent data once enough recovery symbols arrive.

Kernel buffers absorb finite stalls. Application queues absorb downstream variance only while receive continues. Neither fixes sustained throughput below feed rate. Track interface loss, UDP kernel errors, socket overflow, application drops, FEC completion, oldest queued age, and database lag.

Bound each stage. When semantic enrichment falls behind, preserve raw ordered transactions and defer enrichment instead of blocking receive. When persistence falls behind, the system needs an explicit shed or failover policy. UDP will not ask the sender to wait.

Traffic carries source 203.23.178.228. Configure the indexer host to accept it through its chosen UDP socket. Activation remains open for 600 seconds and proves current datagram reachability, not loss-free ingestion or durable indexing.

Parser versions are data

Program layouts and Solana shred formats evolve. Route unknown shred versions and instruction variants to quarantine. Store decoder build and schema version with output. Reprocess retained raw bytes after a parser correction and compare row counts and field changes.

Avoid destructive in-place reinterpretation without an audit trail. A derived table can be rebuilt. The original transaction bytes, slot context, and parser version should remain available for disputed results. Deterministic replay is the indexer's strongest correctness test.

Use fixtures for clean, reordered, duplicated, recovered, and incomplete FEC sets. Test forks and skipped slots. Compare transaction signatures and ordering after canonical commitment. Measure every mismatch instead of assuming an early feed is correct because it is fast.

Serving APIs without overstating certainty

Every response should expose commitment or require it as a query parameter. Streaming clients need retraction or status-update events for provisional data. Cached responses must include commitment in the key. Otherwise an observed row can remain in a cache after its fork disappears.

Freshness service levels should name the start and end points. Packet arrival to provisional API is controllable by the indexer. Leader production to packet arrival is not fully observable without comparable remote timestamps. Provisional API to finalized status is a chain process, not an ingest service promise.

Rate limits and slow consumers must not feed back into packet ingestion. Serve from materialized stores or independent queues. One client requesting a broad address history should never delay shred receive.

Build a two-speed index

Raw shreds let an indexer publish provisional rows within milliseconds, then advance those rows through commitment as canonical state arrives. This design uses a reconciliation pipeline and explicit provenance at every status transition.

Frankfurt serves each indexer destination through its designated network socket. Raw UDP gives the indexer no ordered delivery, gap retransmission, sender backpressure, historical replay, or regional redundancy. Record gaps explicitly and set storage, retention, and recovery policies before ingest begins.

Acceptance is a replay and restart test

Capture a bounded interval and run it through the decoder twice. The resulting ordered transaction identities, parser versions, gap records, and provisional status transitions should match. Then reorder datagrams, insert duplicates, remove recoverable data shreds, and confirm that the canonical decoded output remains the same while recovery metadata changes.

Restart every process at a different boundary. Restart after packet receipt but before decode, after decode but before durable append, and after provisional publication but before commitment. The index should avoid duplicate rows and should never lose the ability to retract a provisional fork. Document which acknowledgement survives each crash.

Compare supported transactions after canonical commitment. A mismatch needs one of four labels: source gap, decoder error, parser scope, or commitment-policy difference. An unlabeled mismatch is not an acceptable residual. Track counts over time and gate deployments when they regress.

Load testing should exceed expected production rate in short bursts and stall persistence for the declared buffer interval. The receiver must expose whether it shed enrichment, raw input, or client delivery. Unbounded queue growth only passes throughput briefly by accumulating latency.

The result is a two-speed index. Milliseconds make provisional records queryable. Seconds advance them toward canonical commitment. Clear provenance lets consumers choose between freshness and certainty.