What is a Solana shred FEC set?

1 set is one recovery problem

An FEC set is the smallest independent Reed-Solomon recovery group in Solana's shred stream. It contains a consecutive run of data shreds and parity shreds computed from equal-length representations of those data shreds.

The boundary matters. A coding shred from one set cannot repair a data shred from another, even inside one shared slot. Their equations were built from different source vectors. Recovery code must partition packets correctly before counting them.

The common FEC set index supplies the group anchor. Coding headers supply dimensions and coding position. Merkle variants also use the FEC set as the authentication tree boundary. This grouping decision therefore controls repair, proof verification, duplicate isolation, memory ownership, and temporary-state release.

The anchor is a data index

Both shred types carry a 32-bit FEC index in their shared header. Its value names the recovery group's initial data-shred index.

If an FEC set index is 128 and its validated data count is 32, the source range is data indexes 128 through 159. The next typical set might begin at 160. Receivers should derive the actual range from the anchor and declared shape, then check that each data member belongs inside it.

The anchor is not a slot-wide ordinal such as set four. It is already expressed in data-index coordinates. This makes data placement direct and lets a receiver group early data packets before a coding header has arrived. It also means arithmetic needs overflow and slot-bound checks before allocation.

Set size is bounded, not universal

Current implementations target groups of 32 data shreds during ordinary shredding. The actual number in a set can differ, especially at a serialization boundary where fewer source shreds remain.

Treating 32 as a permanent wire constant is brittle. Coding headers declare the source and coding counts. Variant rules, implementation limits, and cross-member consistency determine whether those declarations are admissible.

Bounded sets keep Reed-Solomon work manageable and limit how long a single missing packet blocks recovery. They also bound Merkle proof height. A single slot-wide matrix might improve aggregate coding efficiency, but it would delay parity generation, enlarge state, and make one inconsistency affect far more data.

Three numbers define the matrix

Recovery needs an FEC anchor plus separate source and parity counts. Coding position then identifies one parity row. Slot and variant family provide additional grouping and layout context.

A declaration of 20 data and 20 coding shreds creates 40 possible shard positions. Source positions zero through 19 map to data indexes anchor through anchor plus 19. Parity positions map after the source portion in the Reed-Solomon shard array, even though their coding header position starts at zero.

Any 20 independent, valid members are sufficient in the intended erasure model. The receiver can have all 20 data and need no computation, 19 data plus one coding, or 7 data plus 13 coding. The threshold does not depend on which source indexes are absent.

Data arrives before complete metadata

Data headers carry the FEC set index but not the set's data and coding counts. A receiver can therefore see several data shreds before any coding shred declares the matrix dimensions.

Provisional grouping should be sparse and bounded. Store valid data members by their common index under slot and FEC anchor. Do not assume the target size and allocate a full matrix for every unverified packet. When a sanitized coding header arrives, reconcile the provisional members against its declared range.

If no coding shred arrives, complete consecutive data can still be decoded. FEC metadata is required for recovery, not for ordinary ordered assembly. This separation prevents parity delays from holding up a clean data path.

Completion and FEC boundaries differ

DATA_COMPLETE marks the end of a serialized entry vector; an FEC boundary ends a recovery group. The boundaries can coincide, but they describe different layers and are not synonyms.

A completed entry batch may span several FEC groups. Within one group, a data-complete marker may precede the final source if the implementation supports beginning another serialized batch there.

Receivers track both. The FEC lane asks whether enough shards exist to reconstruct source packets. The assembly lane asks whether a consecutive data-index range reaches a completion marker. Advancing either frontier based on the other's metadata can release incomplete bytes or retain already decodable data.

Merkle trees use the same boundary

For Merkle shreds, the leader constructs one canonical tree across every source and parity member. Ordered data leaves come first; ordered coding leaves follow. The signed root commits to the group.

Each packet's branch lets the receiver recompute a root starting at that packet's leaf. Consistent members reach one leader-signed result. If two roots use an identical slot and anchor, they form conflicting views that recovery must keep separate.

This coupling is useful. Erasure reconstruction and authentication refer to the same bounded group. It also means count metadata affects proof interpretation. A receiver must validate dimensions and proof size rather than accepting a branch merely because its final hash has a valid signature.

Recovery begins at the threshold

The source count is the recovery threshold. A 32-source set with 30 data and 2 coding shreds is recoverable. A set with 29 data and 2 coding shreds is not.

Duplicates do not help. Two copies of coding position seven represent the same equation. Conflicting copies cannot both be admitted. The shard count must measure distinct validated source indexes and coding positions within one consistent set view.

Recovery may begin once a useful data range is blocked by a gap and distinct membership reaches the threshold. Waiting for all possible parity adds latency; running with no source gap wastes CPU. A receiver should expose both total unique member count and missing-source count so the trigger remains explicit.

Failure beyond parity capacity

A set with k source shreds and m coding shreds tolerates up to m erasures only when at least k valid total members survive. Loss beyond that threshold leaves multiple possible source vectors consistent with the observed rows.

No local algorithm can infer the missing bytes from insufficient information. The receiver must wait for additional direct or coding shreds, request repair from peers, obtain the block through another source, or abandon the set for its current latency objective.

Authentication does not change the threshold. A valid signature proves a surviving member belongs to a committed view, but it does not generate the missing equations. Likewise, receiving many invalid packets does not improve recovery capacity.

Conflicting metadata needs partitioning

Untrusted UDP can carry two coding shreds with the same slot and FEC anchor but different source counts. It can carry different Merkle roots, repeated positions with different bytes, or data indexes outside the declared range.

Do not let the first packet permanently define a set without evidence. Sanitize each member, authenticate it, and partition conflicts by authenticated commitment where the variant provides one. Set a low bound on concurrent conflicting views and preserve metrics when the bound is exceeded.

Last-write-wins behavior is particularly unsafe. Replacing one coding position after other rows were admitted can make a matrix internally inconsistent. Failed reconstruction must not trigger unlimited retries over arbitrary combinations. Bound attempts and escalate the conflict as protocol evidence.

Memory follows active sets

At sustained feed rates, even short retention creates many live packet objects. FEC grouping adds maps, presence bitmaps, proof state, and optional raw-byte retention.

Use a slot horizon and per-slot set cap. Validate count fields before sizing arrays. Store presence in fixed or bounded bitsets once dimensions are known. Release coding payloads after source completeness unless diagnostics require a brief sample. Keep reconstructed data until the assembly lane consumes it.

Eviction should distinguish complete, unrecoverable, and stale. A complete set can be freed promptly. An unrecoverable set may deserve a short repair window. A future-slot spam set should never displace authenticated current-slot state merely because its numeric slot is larger.

Metrics that expose real loss

Packet-drop counters alone do not describe FEC health. A receiver needs unique data count, unique coding count, recovered source count, threshold wait time, reconstruction duration, invalid member count, and conflict count by set.

Track the first-arrival and recoverable timestamps. Their difference shows how long redundancy took to close a gap. Track when the ordered data frontier consumed the recovered shred. That measures whether FEC was on the critical path for entry decoding.

Do not call every absent data index lost immediately. Reordering is normal. A useful loss metric applies a short observation window or records eventual recovery outcome. Separating reordered, recovered, repaired, and permanently absent packets makes network tuning possible.

FEC state is not consensus state

Completing an FEC set means the receiver has enough authenticated bytes to reconstruct one part of a proposed slot. It says nothing about later votes, fork choice, execution success, or rooting.

The set can belong to a duplicate block. It can be complete while an earlier FEC set in the same slot remains absent. A terminal set can include LAST_IN_SLOT while the receiver still lacks source data in a previous group.

Use names such as set_recoverable, set_data_complete, entry_batch_complete, slot_data_complete, and slot_rooted for separate milestones. These states have different evidence and different consumers. Collapsing them into complete creates race conditions in both trading and archival systems.

A practical set state machine

NEW begins with one sane member. SHAPED follows validated coding metadata that establishes dimensions. RECOVERABLE begins at the source-count threshold of distinct members. SOURCE_COMPLETE means all data positions have arrived or been reconstructed.

CONSUMED means ordered assembly no longer needs the set's data bytes. CONFLICTED records incompatible authenticated claims. EXPIRED records eviction before source completion. A set can move from NEW directly to SOURCE_COMPLETE when all data arrives without coding metadata.

The state machine should be monotonic except for explicit conflict discovery. Late duplicate packets do not reopen consumed state. Late evidence of a conflicting authenticated root should increment a durable diagnostic signal even if the strategy path has already discarded the old payloads.

Path diversity changes set outcomes

FEC success depends on which distinct positions survive, not only the aggregate packet-loss percentage. Losing 20 packets spread across 20 different sets may require one parity shard in each. Losing the same 20 packets inside a 16-plus-16 set can make that one set unrecoverable.

Multiple delivery paths can reduce correlated erasures when their loss domains differ. Duplicate traffic adds bandwidth, but a second copy arriving through the same congested queue may add little independence. Record transport path beside shred identity before deduplication so operators can measure which path supplied the first valid copy and which path closed a recovery threshold.

Set-level histograms expose concentration. Track missing-source count at first completion marker, members available at recovery time, and unrecoverable sets by slot. One global loss rate can look healthy while a few concentrated gaps block the transaction batches with the highest latency value.