What is the difference between data and coding shreds?

2 types, one delivery stream

Solana sends both shred types through one propagation system. They share a header prefix and packet-sized transport shape, but perform different jobs.

Data packets supply source material. Concatenating their meaningful payloads produces serialized entry vectors, which record Proof of History progress alongside transactions. Coding packets supply redundancy: their payloads are Reed-Solomon equations calculated over normalized source representations within a single recovery group.

A raw UDP receiver should classify the variant before doing anything with the tail. Feeding a coding payload to an entry decoder produces noise. Discarding every coding shred saves bandwidth inside the process but removes the ability to repair ordinary packet loss. Consumer policy follows its objective: earliest decodable prefix, eventual local completeness, or both.

Data shreds carry the ledger stream

The leader serializes entries, then cuts that stream into packet-sized data regions. Every data packet reports slot, sequential data index, parent distance, completion state, and a boundary for meaningful bytes.

Data index determines concatenation order. Arrival order does not. Index 41 belongs after index 40 even if it reached the receiver first. DATA_COMPLETE closes a consecutive range that can be deserialized as an entry vector. The proposed slot's terminal range carries LAST_IN_SLOT.

Payload boundaries ignore transaction framing. A transaction may cross packets, while several small transactions may fit in one. Headers still permit early placement, but safe content decoding requires a complete serialized range or explicit partial-framing state.

Coding shreds carry equations

A coding shred contains no independent copy of a lost packet. It contains one parity shard, an equation over all source shards in its FEC set. The coding header declares source count, parity count, and this parity shard's position.

With 32 source members, every byte column contributes 32 source symbols. Reed-Solomon encoding evaluates additional independent combinations to produce parity symbols, repeating across the shard length. Enough independent surviving rows let the receiver solve for missing source values.

The practical threshold is source count, not a fixed number of coding packets. In a 32 plus 32 set, any 32 valid and mutually consistent shards can recover all 32 data shreds. Thirty-one distinct shards cannot, even if the receiver has hundreds of duplicate datagrams.

Headers differ after byte 83

Each type starts with the same 83-byte prefix. It carries signature, variant, slot, index, version, and recovery anchor. The next fields diverge.

Data then adds its ancestry distance, completion flags, and meaningful size. Coding instead declares how many source and parity members exist, followed by its own parity position. These fields describe recovery geometry.

The resulting current header prefixes are 88 bytes for data and 89 bytes for coding. That one-byte difference is less important than the semantic split. Parsing all packets with one struct shifts every field after the common header and can create integers that look reasonable while referring to unrelated bytes.

Index and position are different coordinates

Data shred indexes increase through the slot and define ledger order. Coding shreds also carry common indexes in their own type namespace. In addition, each coding shred has a position within its FEC set.

Recovery code derives source indexes relative to the FEC anchor, while parity members are placed by coding position. A common coding index never maps directly to a missing data index. Since every parity row combines the full source group, any position can help reconstruct any absent source shard.

Storage should retain packet identity using common index together with type and slot. Recovery state instead uses a slot plus recovery anchor, validates dimensions, and places parity bytes by position. Combining both maps into one integer key makes boundary groups and conflicts harder to reason about.

FEC sets bind both types

One FEC set forms the unit for erasure recovery and Merkle authentication. Consecutive source packets belong to the group alongside their corresponding parity collection.

Current production uses bounded groups rather than one slot-wide erasure matrix. This bounds encoding work, proof size, buffering, and recovery latency. A batch may end with fewer source packets, so receivers must sanitize declared counts instead of enforcing one universal group size.

All admitted members must agree on slot, FEC set anchor, dimensions, variant family, and authentication commitments. A coding shred from the same slot but another FEC anchor is mathematically unrelated. Adding it to the shard count creates false confidence and will lead to failed reconstruction or corrupt output.

Why Solana sends parity

UDP can drop packets without notifying either endpoint. Turbine also distributes different shreds through different paths, so loss can affect a subset of a validator's view. Waiting for a request and retransmission adds at least one network round trip and can concentrate repair load.

Forward error correction moves recovery bytes proactively. A receiver missing data index 109 can recover it as soon as enough other members of its FEC set arrive, without identifying a repair peer and waiting for that peer to respond.

The cost is substantial bandwidth and encoding work. Near-equal source and parity counts can almost double source traffic before other overhead. Solana accepts that cost because predictable parallel delivery and local repair matter more than minimizing every transmitted byte.

Recovery does not always wait

A receiver that has every data shred needed for a completed range can decode immediately. Coding shreds are unnecessary on that fast path. Recovery starts only when a gap blocks useful progress and enough total members are present.

This suggests two lanes. The ordered data lane advances a contiguous frontier and emits complete entry batches. The FEC lane accumulates data and coding shards, reconstructs gaps, sanitizes recovered shreds, and feeds them back into the data lane.

Running Reed-Solomon for every set on arrival wastes CPU when source data is already complete. Waiting until the whole slot ends wastes latency when one missing packet blocks a batch. Trigger recovery based on a known gap, a completion marker beyond that gap, and sufficient distinct shards.

Coding-first arrival is normal

Network scheduling does not promise that source packets beat parity packets. A coding shred can be the first observed member of a set. It may even provide the first reliable declaration of set dimensions.

Receivers should create bounded provisional state keyed by slot and FEC anchor. They can validate the coding header, place its parity position, and wait for data. They should not interpret coding-first arrival as loss by itself. A direct data copy may arrive microseconds later on another Turbine path.

Timeouts must reflect the goal. A trading parser may abandon old incomplete state quickly after the opportunity window closes. A ledger archival path may retain it longer and request repair. Both policies can share parsing and FEC code while using different eviction horizons.

Recovered data needs full validation

Reed-Solomon output is a byte array, not automatically an admitted shred. The receiver must reconstruct the proper packet form, parse the data header, validate the index and FEC anchor, check size and flags, and confirm authentication consistency.

Merkle recovery adds a relationship among reconstructed leaf data, proof material, and signed root. Implementations should follow the protocol recovery routine precisely. Sending parity output around normal shred sanitization creates a privileged path for malformed group metadata.

Recovered shreds should be observable as recovered. That label helps explain latency and loss. It should not change their data semantics after validation. Ordered assembly can consume a valid recovered data shred at index 109 in the same position as a directly received copy.

Duplicate and conflicting shreds

Multi-path systems routinely deliver identical bytes more than once. Deduplicate using authenticated identity plus content. Repeated copies add nothing to the independent shard count.

Conflicts require another policy. Different byte strings under one slot/type/index claim may indicate corruption, spam, or duplicate leader production. Last-write-wins replacement can combine incompatible block proposals inside one FEC matrix.

Merkle roots help partition conflicting FEC-set views. Legacy signatures authenticate individual packet bodies, so consistency still requires careful set tracking. Preserve bounded evidence, increment conflict metrics, and never mix roots or structural declarations merely because slot and recovery anchor match.

Bandwidth and memory consequences

The shredstream.sh feed carries source data alongside recovery traffic. Its sustained rate varies as network activity changes, so capacity planning must include both types.

Consider a receiver observing 6,750 packets each second. Retaining one second creates about 6,750 packet objects before allocator overhead, indexes, signatures, and duplicates. Ten seconds of raw history can therefore consume far more memory than a payload-byte calculation suggests.

Separate hot fields from cold bytes. Parse identity and recovery coordinates into compact structures. Keep raw payloads in bounded pools. Release coding shards once the set is complete and outside any diagnostic retention window. Memory policy must also resist untrusted future-slot traffic and count declarations.

Choosing a consumer policy

An event-driven trading system usually wants the earliest valid completed entry batch. It takes consecutive data immediately, invokes FEC only when a gap blocks a marked boundary, and treats later consensus status as reconciliation.

A validator-like consumer needs slot completeness and longer repair behavior. An observability collector may want all coding traffic because parity arrival reveals path loss even when data arrived elsewhere. A protocol analyzer may retain conflicts and proofs longer than either production path.

No one policy dominates. Dropping coding shreds reduces CPU and memory but turns a single data loss into a decoding gap. Retaining all coding shreds improves repair odds but expands the attack surface for count, position, and allocation abuse. State the objective before tuning the pipeline.

Data is early, not settled

Data packets expose a ledger proposal before RPC can report executed state. Coding packets increase the chance of reconstructing that early proposal despite network loss. Neither type supplies consensus finality.

A leader can produce a slot that is later abandoned. An embedded transaction can fail execution. A valid coding set can recover bytes for a fork that never roots. Conversely, local gaps reveal nothing by themselves about network acceptance of the block.

Downstream schemas should carry slot, receive time, authentication state, recovery state, and later commitment state separately. That vocabulary lets strategy code act on low-latency evidence without treating either shred type as canonical block state.

Type-aware queues prevent head-of-line blocking

Both packet types can enter one socket and then take separate work paths. The receiver parses their shared prefix once, routing admitted source data to ordered assembly and admitted parity to FEC storage.

Separate bounded queues let the system prioritize a data shred that advances the current decoding frontier without discarding parity needed for an active gap. One undifferentiated FIFO can place hundreds of surplus coding packets for completed sets ahead of a source packet that closes a transaction batch.

Priority is an operational choice, not a validity rule. Every packet still receives safe header checks. Under pressure, identical duplicates and parity for source-complete sets are lower-value work than authenticated frontier data or recoverable-gap parity. Metrics should record these policy drops separately from network loss and malformed rejection.