Data shreds and coding shreds

Data carries ledger bytes

A data shred combines the shared header, its data-specific header, serialized ledger bytes, and variant-dependent Merkle material. Its type-specific header supplies parent offset, flags, and the declared data size. Obtain ledger bytes through the typed data() accessor. Exclude headers, padding, proof, chained root, and retransmitter signature from the deshredded stream.

Within each slot, data-shred indexes increase monotonically. Order shreds on this index. Completion flags divide that sequence into consecutive serialized units. DATA_COMPLETE_SHRED ends one unit. The proposed slot end is marked by LAST_SHRED_IN_SLOT, which also implies data complete.

A transaction is not aligned to a shred. Several data shreds may be needed for one serialized entry batch. Conversely, one shred can finish an earlier entry and begin the next. Do not scan individual shred payloads for transaction structures.

Coding carries parity

A coding shred combines common and coding headers with Reed-Solomon parity bytes and Merkle material. Its header supplies position in 0..m, with num_data_shreds defining k and num_coding_shreds defining m. Its local erasure-shard index is k + position.

Coding parity covers the erasure-coded representation of entire data shreds, including headers required to reconstruct a missing data member. Coding headers themselves are not part of the data shard stream. The parity slice and data erasure slice are equal length after the protocol's variant-dependent restrictions.

Do not append coding payloads to ledger data. Use them only when one or more data shards in the same FEC set are missing. If every data shred is present, the coding members can be discarded after integrity and duplicate accounting.

Keep index spaces separate

The common index field means a data index on data shreds and a coding index on coding shreds. A unique identity is:

#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]enum ShredType { Data, Code } #[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]struct ShredId { slot: u64, index: u32, kind: ShredType } #[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]struct FecKey { slot: u64, fec_set_index: u32 } fn code_local_index(k: u16, position: u16, m: u16) -> Result<usize, &'static str> {    if position >= m { return Err("coding position outside parity range"); }    Ok(usize::from(k) + usize::from(position))} fn main() {    if (code_local_index(32, 3, 32) == Ok(35)) == false { std::process::exit(1); }}

Do not use index / 32 to derive a set. The explicit fec_set_index field identifies the set. Current construction typically targets 32 members of each type, data and coding. A shorter final set is valid, and future releases can change construction.

Determine data positions

Within an FEC set, a data shred at global data index i maps to local erasure index i - fec_set_index. Subtract with an underflow check. Once coding configuration is known, also require that result in the range below k. A data index below the set start is invalid. A member beyond the declared k is a conflicting header or wrong set.

You may receive data before any coding shred, so k and m can be unknown initially. Store sanitized data members by global index under the FEC key. When a coding shred arrives, validate every stored member against its declared range. Bound pre-configuration state by count, bytes, and time.

Decide when to recover

Let present count unique valid shards of both types in a compatible set. Reed-Solomon recovery becomes possible at present >= k. Invoke it only if data is missing. A total loss above m is unrecoverable because fewer than k shards remain.

Receiving k coding shreds without any data can mathematically reconstruct data if the configured set has at least k parity shards. In the usual 32 by 32 set it can. The implementation still needs compatible authenticated members and correct erasure slices.

After reconstruction, parse and sanitize each recovered data shred, then verify its identity and FEC key. Rebuild the Merkle tree and compare that result with the root authenticated by received members. Reed-Solomon output alone is not authenticated.

Prioritize work

Forward verified data shreds immediately to FEC ownership and ordering state. Coding shreds should update the same FEC set but need not block already complete data. Under application memory pressure, discard coding members only for sets whose full data is already accepted. Never discard parity for an incomplete set without counting the reduced recovery margin.

Duplicate handling must compare canonical content. The same ShredId with equal authenticated content is a duplicate. The same ID with different content is a conflict. Retransmitter signatures may differ over an otherwise identical leader-authenticated shred. Use pinned Agave duplicate comparison, or omit only its retransmitter-signature region as defined by the format.

Monitor both paths

Count input by data or coding, unique accepted members, duplicates, conflicting IDs, complete-data bypass, recovery attempts, recovered data count, and unrecoverable sets. A coding-to-data ratio that changes sharply can indicate capture filters, parser regression, or a network release. It is not by itself proof of a service fault because final FEC sets and traffic timing vary.