Data shreds and coding shreds

Data carries ledger bytes

A data shred contains a common header, data header, serialized ledger data, and variant-dependent Merkle material. The data header supplies a parent offset, flags, and declared data size. Call the typed data() accessor to obtain ledger bytes. Do not include header, padding, proof, chained root, or retransmitter signature in the deshredded stream.

Data shreds have a monotonically increasing data index within a slot. Order them by that index. Completion flags divide the data-index sequence into consecutive serialized units. DATA_COMPLETE_SHRED marks the end of one unit. LAST_SHRED_IN_SLOT also implies data complete and marks the proposed end of the slot.

A transaction is not aligned to a shred. One serialized entry batch can span many data shreds, and one data shred can contain portions of several entries. Do not scan individual shred payloads for transaction structures.

Coding carries parity

A coding shred contains a common header, coding header, Reed-Solomon parity bytes, and Merkle material. Its coding header declares num_data_shreds as k, num_coding_shreds as m, and position in 0..m. Its local erasure-shard index is k + position.

Coding parity covers the erasure-coded representation of entire data shreds, including headers required to reconstruct a missing data member. Coding headers themselves are not part of the data shard stream. The parity slice and data erasure slice are equal length after the protocol's variant-dependent restrictions.

Do not append coding payloads to ledger data. Use them only when one or more data shards in the same FEC set are missing. If every data shred is present, the coding members can be discarded after integrity and duplicate accounting.

Keep index spaces separate

The common index field means a data index on data shreds and a coding index on coding shreds. A unique identity is:

#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
enum ShredType { Data, Code }

#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
struct ShredId { slot: u64, index: u32, kind: ShredType }

#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
struct FecKey { slot: u64, fec_set_index: u32 }

fn code_local_index(k: u16, position: u16, m: u16) -> Result<usize, &'static str> {
    if position >= m { return Err("coding position outside parity range"); }
    Ok(usize::from(k) + usize::from(position))
}

fn main() {
    if (code_local_index(32, 3, 32) == Ok(35)) == false { std::process::exit(1); }
}

Do not use index / 32 to derive a set. The explicit fec_set_index field identifies the set. The current typical target uses 32 data and 32 coding shreds, but a final set can be shorter and future releases can change construction.

Determine data positions

Within an FEC set, a data shred at global data index i maps to local erasure index i - fec_set_index. Require checked subtraction and require the result to be less than k after coding configuration is known. A data index below the set start is invalid. A member beyond the declared k is a conflicting header or wrong set.

You may receive data before any coding shred, so k and m can be unknown initially. Store sanitized data members by global index under the FEC key. When a coding shred arrives, validate every stored member against its declared range. Bound pre-configuration state by count, bytes, and time.

Decide when to recover

Let present be the number of unique valid data and coding shards for a compatible set. Reed-Solomon recovery is possible when present >= k. Run it only when at least one data shard is missing. More than m missing total shards is unrecoverable because fewer than k remain.

Receiving k coding shreds without any data can mathematically reconstruct data if the configured set has at least k parity shards. In the usual 32 by 32 set it can. The implementation still needs compatible authenticated members and correct erasure slices.

After reconstruction, parse and sanitize each recovered data shred, verify its identity and FEC key, rebuild the Merkle tree, and require the root to equal the root authenticated by received members. Do not treat Reed-Solomon output as authenticated on its own.

Prioritize work

Forward verified data shreds immediately to FEC ownership and ordering state. Coding shreds should update the same FEC set but need not block already complete data. Under application memory pressure, discard coding members only for sets whose full data is already accepted. Never discard parity for an incomplete set without counting the reduced recovery margin.

Duplicate handling must compare canonical content. The same ShredId with equal authenticated content is a duplicate. The same ID with different content is a conflict. Retransmitter signatures can differ while the underlying leader-authenticated shred is the same, so use the pinned Agave duplicate comparison or exclude only the format-defined retransmitter signature region.

Monitor both paths

Count input by data or coding, unique accepted members, duplicates, conflicting IDs, complete-data bypass, recovery attempts, recovered data count, and unrecoverable sets. A coding-to-data ratio that changes sharply can indicate capture filters, parser regression, or a network release. It is not by itself proof of a service fault because final FEC sets and traffic timing vary.