What is the difference between data and coding shreds?

2 types, one delivery stream

Solana emits data shreds and coding shreds into the same propagation system. They share a common header and packet-sized transport shape, but they serve different jobs.

Data shreds are source material. Their meaningful payload bytes concatenate into serialized vectors of entries. Entries contain transactions and Proof of History progress. Coding shreds are redundancy. Their payload bytes are Reed-Solomon parity derived from equal-length representations of the source data shreds in one FEC set.

A raw UDP receiver should classify the variant before doing anything with the tail. Feeding a coding payload to an entry decoder produces noise. Discarding every coding shred saves bandwidth inside the process but removes the ability to repair ordinary packet loss. The correct choice depends on whether the consumer values the earliest decodable prefix, eventual local completeness, or both.

Data shreds carry the ledger stream

The leader serializes entries and divides that byte stream into packet-sized data regions. Each data shred carries a slot, a sequential data index, a parent offset, completion flags, and a size that bounds meaningful bytes.

Data index determines concatenation order. Arrival order does not. Index 41 belongs after index 40 even if it reached the receiver first. The data-complete flag marks a boundary at which a consecutive range can be deserialized as an entry vector. The last-in-slot flag marks the terminal range for the proposed slot.

The payload is not aligned to transactions. One transaction can cross a shred boundary. Several small transactions can share one shred. Headers can therefore drive early placement, but content decoding still waits for a complete serialized range or a decoder with safe partial framing state.

Coding shreds carry equations

A coding shred contains no independent copy of a lost packet. It contains one parity shard, an equation over all source shards in its FEC set. The coding header declares source count, parity count, and this parity shard's position.

If a set contains 32 data shreds, each byte column across those shreds forms 32 source symbols. Reed-Solomon encoding evaluates additional independent combinations to produce coding symbols. The process repeats across the shard length. Enough independent surviving rows let the receiver solve for missing source symbols.

The practical threshold is source count, not a fixed number of coding packets. In a 32 plus 32 set, any 32 valid and mutually consistent shards can recover all 32 data shreds. Thirty-one distinct shards cannot, even if the receiver has hundreds of duplicate datagrams.

Headers differ after byte 83

Both types begin with the 83-byte common header: signature, variant, slot, index, version, and FEC set index. Their next fields diverge.

A data shred adds parent offset, flags, and size. Those fields describe ledger ancestry, serialization boundaries, and meaningful payload length. A coding shred adds number of data shreds, number of coding shreds, and coding position. Those fields describe recovery geometry.

The resulting current header prefixes are 88 bytes for data and 89 bytes for coding. That one-byte difference is less important than the semantic split. Parsing all packets with one struct shifts every field after the common header and can create integers that look reasonable while referring to unrelated bytes.

Index and position are different coordinates

Data shred indexes increase through the slot and define ledger order. Coding shreds also carry common indexes in their own type namespace. In addition, each coding shred has a position within its FEC set.

Recovery code maps source data indexes from the FEC set anchor and maps parity shards from coding position. It does not map a coding common index directly onto a missing data index. One parity position can help reconstruct any missing source shard because each parity row combines the entire source set.

Storage code should retain slot, type, and common index as packet identity. Recovery code should group by slot and FEC set index, validate declared dimensions, then place coding bytes by position. Combining these two maps into one clever integer key usually makes boundary sets and conflicts harder to reason about.

FEC sets bind both types

An FEC set is the unit of erasure recovery and Merkle authentication. It contains a run of data shreds and a corresponding collection of coding shreds.

Current production aims for bounded groups rather than one erasure matrix covering an entire slot. Bounded groups control encoding work, proof size, buffering, and recovery latency. The final group in a batch can contain fewer source shreds, so receivers should read and validate declared counts instead of requiring one universal group size.

All admitted members must agree on slot, FEC set anchor, dimensions, variant family, and authentication commitments. A coding shred from the same slot but another FEC anchor is mathematically unrelated. Adding it to the shard count creates false confidence and will lead to failed reconstruction or corrupt output.

Why Solana sends parity

UDP can drop packets without notifying either endpoint. Turbine also distributes different shreds through different paths, so loss can affect a subset of a validator's view. Waiting for a request and retransmission adds at least one network round trip and can concentrate repair load.

Forward error correction moves recovery bytes proactively. A receiver missing data index 109 can recover it as soon as enough other members of its FEC set arrive, without identifying a repair peer and waiting for that peer to respond.

The cost is substantial bandwidth and encoding work. A roughly one-to-one data and coding ratio can approach twice the source traffic before other overhead. Solana accepts that cost because predictable parallel delivery and local repair matter more than minimizing every transmitted byte.

Recovery does not always wait

A receiver that has every data shred needed for a completed range can decode immediately. Coding shreds are unnecessary on that fast path. Recovery starts only when a gap blocks useful progress and enough total members are present.

This suggests two lanes. The ordered data lane advances a contiguous frontier and emits complete entry batches. The FEC lane accumulates data and coding shards, reconstructs gaps, sanitizes recovered shreds, and feeds them back into the data lane.

Running Reed-Solomon for every set on arrival wastes CPU when source data is already complete. Waiting until the whole slot ends wastes latency when one missing packet blocks a batch. Trigger recovery based on a known gap, a completion marker beyond that gap, and sufficient distinct shards.

Coding-first arrival is normal

Network scheduling does not promise that source packets beat parity packets. A coding shred can be the first observed member of a set. It may even provide the first reliable declaration of set dimensions.

Receivers should create bounded provisional state keyed by slot and FEC anchor. They can validate the coding header, place its parity position, and wait for data. They should not interpret coding-first arrival as loss by itself. A direct data copy may arrive microseconds later on another Turbine path.

Timeouts must reflect the goal. A trading parser may abandon old incomplete state quickly after the opportunity window closes. A ledger archival path may retain it longer and request repair. Both policies can share parsing and FEC code while using different eviction horizons.

Recovered data needs full validation

Reed-Solomon output is a byte array, not automatically an admitted shred. The receiver must reconstruct the proper packet form, parse the data header, validate the index and FEC anchor, check size and flags, and confirm authentication consistency.

Merkle recovery has an additional relationship between reconstructed leaf data, proof material, and the signed root. Implementations should use the protocol's recovery routine or match it carefully. Taking parity output and bypassing ordinary shred sanitization creates a privileged path for malformed set metadata.

Recovered shreds should be observable as recovered. That label helps explain latency and loss. It should not change their data semantics after validation. Ordered assembly can consume a valid recovered data shred at index 109 in the same position as a directly received copy.

Duplicate and conflicting shreds

Duplicate delivery of identical bytes is expected in multi-path systems. Deduplicate by authenticated shred identity and content. Repeated packets do not increase the independent shard count.

Conflicting packets are different. Two byte strings claiming the same slot, type, and index can indicate corruption, spam, or duplicate leader production. Last-write-wins replacement can combine source shreds from incompatible blocks inside one FEC matrix.

Merkle roots help partition conflicting FEC-set views. Legacy signatures authenticate individual packet bodies, so consistency still requires careful set tracking. Preserve bounded evidence, increment conflict metrics, and avoid mixing roots or structural declarations even when the slot and FEC anchor match.

Bandwidth and memory consequences

The measured shredstream.sh feed averages 54.3 Mbps, 5,585 packets per second, and 1,216 bytes per packet over its stated observation window. Those totals include both useful source data and recovery traffic.

A process that retains every packet for one second sees about 5,585 packet objects before accounting for allocator overhead, indexes, signatures, and duplicate copies. Keeping ten seconds of raw packet history can consume far more memory than multiplying datagram bytes alone suggests.

Separate hot fields from cold bytes. Parse identity and recovery coordinates into compact structures. Keep raw payloads in bounded pools. Release coding shards once the set is complete and outside any diagnostic retention window. Memory policy must also resist untrusted future-slot traffic and count declarations.

Choosing a consumer policy

An event-driven trading system usually wants the earliest valid completed entry batch. It takes consecutive data immediately, invokes FEC only when a gap blocks a marked boundary, and treats later consensus status as reconciliation.

A validator-like consumer needs slot completeness and longer repair behavior. An observability collector may want all coding traffic because parity arrival reveals path loss even when data arrived elsewhere. A protocol analyzer may retain conflicts and proofs longer than either production path.

No one policy dominates. Dropping coding shreds reduces CPU and memory but turns a single data loss into a decoding gap. Retaining all coding shreds improves repair odds but expands the attack surface for count, position, and allocation abuse. State the objective before tuning the pipeline.

Data is early, not settled

Data shreds expose proposed ledger content earlier than executed RPC state. Coding shreds improve the probability that a receiver can reconstruct that proposal despite network loss. Neither type supplies consensus finality.

A leader can produce a slot that is later abandoned. An embedded transaction can fail execution. A valid coding set can recover bytes for a fork that never roots. Conversely, missing local shreds do not prove that the network rejected the block.

Downstream schemas should carry slot, receive time, authentication state, recovery state, and later commitment state separately. That vocabulary lets strategy code act on low-latency evidence without pretending that data and coding shreds have already become a canonical block.

Type-aware queues prevent head-of-line blocking

Data and coding packets can share one socket while taking different work paths. A receiver can parse the common header once, then direct admitted data toward ordered assembly and admitted coding toward FEC storage.

Separate bounded queues let the system prioritize a data shred that advances the current decoding frontier without discarding parity needed for an active gap. One undifferentiated FIFO can place hundreds of surplus coding packets for completed sets ahead of a source packet that closes a transaction batch.

Priority is an operational choice, not a validity rule. Every packet still receives safe header checks. Under pressure, identical duplicates and parity for source-complete sets are lower-value work than authenticated current-frontier data or coding for a blocked recoverable set. Metrics should record these policy drops separately from network loss and malformed rejection.