How does erasure coding work in Solana?

k source shards define the threshold

Reed-Solomon erasure coding starts with k equal-length source shards and computes m parity shards. Solana's source shards correspond to data shreds within one FEC set. Its parity shards travel as coding shreds.

The defining property is threshold recovery. Any k independent, valid shards from the k plus m total can reconstruct all k sources. A 32 plus 32 group can lose any 32 packets and remain recoverable, provided the other 32 belong to one consistent set.

This is erasure correction, not general error correction. The receiver must know which positions are absent or invalid. Authentication and sanitization identify unusable packets. Reed-Solomon then fills known gaps. Feeding unknown corrupt bytes into the matrix as though they were valid can produce corrupt output rather than a helpful rejection.

Systematic means data remains data

The code is systematic. Original data shards are transmitted unchanged as data shreds, alongside computed coding shreds. A receiver with every required data shred does not need to decode parity.

This property preserves the fastest path. Consecutive data can move directly into ordered assembly and entry deserialization. The coding path stays idle until loss blocks progress. A non-systematic code would require decoding even when every packet arrived, adding work to the common case.

Systematic does not mean the packet header is outside the code. The erasure representation includes protocol-defined portions of each data shred so recovery can recreate a structurally valid source packet. The exact byte range differs between legacy and Merkle formats and should come from variant-aware code.

Equal lengths are mandatory

Reed-Solomon operates across equal-length shards. Each byte offset forms a column of source symbols, and parity is computed for that column. Unequal source lengths would leave matrix rows undefined after a shorter shard ended.

Solana pads unused source capacity with zero bytes. The data size field still marks meaningful ledger bytes. Padding participates in the coding representation where the variant requires it, but it does not become part of the serialized entry stream.

Receivers should never right-pad a truncated UDP datagram and call it a valid shard. Protocol padding is authenticated and generated before transmission. Network truncation loses unknown bytes. A short datagram must fail layout validation rather than enter recovery as a zero-extended source.

Parity is a matrix of equations

At each byte position, k source symbols form a vector. A generator matrix maps that vector to k systematic rows and m parity rows. The selected surviving rows form a square system when k shards are available.

Recovery inverts that surviving-row system in a finite field and solves for the original source symbols. Implementations cache coding structures for common dimensions because building matrices repeatedly costs more than applying an existing configuration.

The mathematics explains why positions matter. Two copies of one coding position are the same row, so they add no rank. Twenty distinct packet objects are not enough for a 20-source set if three are duplicates of existing positions. Count unique, validated coordinates.

The encoded byte range is variant-aware

Legacy coding covers the data shred representation required to recreate headers and payload in the legacy layout. Merkle coding uses the packet material defined as the erasure shard while proof bytes remain variant-specific authentication material.

The distinction prevents a circular dependency. Merkle proofs authenticate membership in a tree built over set leaves, while parity supports reconstruction of source leaf material. Protocol code knows which tail regions are proof, chaining, or optional signature data and excludes or handles them accordingly.

Handwritten decoders often fail here. A slice that works for legacy packets can be off by the Merkle proof length. Recovery may still return bytes of the expected length, but their fields will not sanitize. Keep erasure-shard extraction and packet reassembly behind variant-specific implementations.

Missing and corrupt are different inputs

An erasure is a position known to be unavailable. A corruption is a supplied position containing wrong bytes. Standard recovery succeeds reliably only after corrupt packets have been excluded.

Shred signatures, Merkle proofs, variant validation, count checks, and root consistency turn suspicious packets into erasures. UDP checksums and local datagram lengths help detect transport damage but are not sufficient for protocol attribution.

If a packet fails authentication, mark its coordinate absent for that set view. Do not retain its payload as a tentative equation because the matrix routine accepts bytes without understanding signatures. A false row can contaminate every reconstructed source shard.

Recovery is local and parallel

Each FEC set can be recovered independently. Sets from different slots and different anchors can run on separate worker queues. This fits Solana's broader propagation model, where shreds arrive through many network paths without one ordered connection.

Local recovery avoids a repair round trip. Once the threshold arrives, CPU and memory bandwidth are the remaining costs. For a latency-sensitive receiver, a warm Reed-Solomon cache and bounded worker handoff can reconstruct a small number of gaps before a network request could return.

Parallelism needs admission control. An attacker can send many fake anchors or count combinations. Cheap parsing, version filtering, leader attribution, and per-slot limits should happen before expensive matrix work or unbounded task creation.

Recovery should be demand-driven

Complete source data requires no parity work. Incomplete source data cannot always benefit immediately. A receiver should invoke recovery when three conditions hold: a known data gap blocks a useful boundary, the set shape is validated, and unique consistent members meet the source threshold.

This avoids decoding every set twice, once speculatively and once after late data arrives. It also avoids holding an entry batch after parity has already made the gap repairable.

The trigger can be expressed as missing_source_count greater than zero, unique_member_count at least data_count, and an assembly frontier or completion marker waiting beyond the gap. Archival consumers can recover more eagerly. Trading consumers tend to prioritize gaps on the current decoding frontier.

Reconstructed packets reenter validation

Matrix output reconstructs source shard bytes. The receiver then rebuilds the data shred representation required by the active variant. The result must pass the same structural checks as a directly received data shred.

Validate slot, data index, FEC anchor, parent offset, flags, size, and authentication relationship. Check that a recovered packet does not conflict with an authenticated direct copy that arrived during computation. Only then insert it into ordered assembly.

This second pass is not redundant. Malformed but authenticated set metadata, implementation bugs, and mixed set views can all produce bytes that need rejection. Recovery is one transformation inside the trust pipeline, not a privilege escalation around it.

More parity trades bandwidth for tolerance

m coding shreds add capacity to survive up to m erasures while at least k total valid rows remain. More parity increases network traffic, serialization, signing or proof work, memory, and receiver admission load.

Less parity lowers those costs but makes ordinary path loss more likely to require explicit repair. The right ratio depends on propagation topology and protocol policy, not one receiver's local preference. A raw subscriber consumes the ratios the leader and network produce.

At 54.3 Mbps measured feed bandwidth, parity is not an abstract cost. Socket buffers, network interfaces, packet parsers, and observability systems must handle the combined stream. Filtering coding traffic at the application edge reduces later work but does not reduce ingress traffic already delivered.

Recovery latency has several clocks

First data arrival starts the opportunity clock. Gap discovery happens when a later index or completion marker arrives. Threshold time occurs when enough distinct set members are present. Matrix duration covers reconstruction. Validation and ordered insertion follow.

Measure each point with a monotonic clock. A single recovery_ms metric hides whether delay came from network loss, late parity, worker queueing, matrix computation, or a blocked assembly lane.

A recovery taking 60 microseconds is irrelevant if the threshold arrived 8 milliseconds after the completed boundary. Conversely, fast parity arrival can still miss a strategy deadline if recovery sat behind bulk historical work. Queue priority should reflect active slot and blocked frontier, not arrival order alone.

When Reed-Solomon cannot help

Fewer than k valid unique members leave the source underdetermined. No decoder setting can recover information that was not received. The consumer must wait, repair, use another feed, or abandon the latency window.

Reed-Solomon also cannot decide between conflicting authenticated roots. Those are distinct proposed data sets, not random packet erasures. It cannot establish which fork consensus will select. It cannot repair a parser that extracted the wrong byte range.

Coding protects against loss within its designed model. It does not replace authentication, duplicate handling, fork tracking, or protocol-version support. Systems that describe it as guaranteed delivery hide the exact failure modes operators need to see.

Implementation discipline matters

Use a well-tested finite-field library or the same recovery routines as the supported validator code. Cache validated configurations by data and coding counts. Pool shard buffers at bounded sizes. Clear or overwrite reused buffers so stale bytes do not enter short reconstructions.

Test every erasure pattern near the tolerance boundary. Include all-data, all-parity subsets where supported by dimensions, scattered losses, consecutive losses, duplicates, conflicts, truncated packets, and final partial sets. Verify reconstructed bytes against original serialized shreds before testing only higher-level entries.

Operationally, expose recovery failures by reason. Too few shards, inconsistent dimensions, invalid position, root conflict, matrix error, and reconstructed-sanitize failure point to different causes. One generic FEC failed counter makes diagnosis guesswork.

Selecting rows is a correctness decision

A recoverable set can contain more than k members. The decoder does not need all of them for one matrix solve, but its selection must include distinct positions from one authenticated view. A deterministic preference for present source rows followed by low coding positions makes behavior reproducible and minimizes reconstruction work.

Extra members remain useful for validation. If reconstruction from one k-row subset produces source bytes that disagree with authenticated direct data or cannot satisfy the committed Merkle root, the set is inconsistent. Retrying every possible subset is combinatorial and creates an attacker-controlled CPU loop. Bound alternate attempts and report the conflict.

Sources that arrive during recovery create a race. Snapshot the presence map and view identifier when scheduling work. At completion, compare the result with any newly admitted direct source. Identical bytes confirm the recovery. Different authenticated bytes escalate the set rather than allowing whichever write acquires a lock last.

The matrix cache also needs a trusted key. Data count and coding count select dimensions, but variant-aware shard length remains part of buffer handling. An entry for a 32-plus-32 configuration should cache finite-field coefficients, not untrusted packet pointers or proof state.

Deterministic selection, bounded retries, and post-recovery comparison make failures reproducible. Without them, two receiver processes can choose different mixtures from the same conflict and emit different proposed transactions even though both report FEC success.

Keep the selected row coordinates in trace data. A later byte mismatch can then be reproduced from the same subset instead of inferred from aggregate set counters.