Verify Merkle shreds

Verify membership and authority

A Merkle shred carries one leaf's material and a proof leading to an FEC-set root. The leader's Ed25519 signature authenticates that root. Verification therefore has two linked results: the member bytes must reconstruct a root through the proof, and the signature must verify over that root with the scheduled leader key.

Calling only Ed25519 over packet bytes is wrong for current Merkle variants. Checking only the Merkle proof is also insufficient because anyone can construct a self-consistent tree.

Use the pinned Agave implementation:

use solana_ledger::shred::Shred;
use solana_pubkey::Pubkey;

fn verify_merkle(packet: &[u8], leader: &Pubkey) -> Result<Shred, String> {
    let shred = Shred::new_from_serialized_shred(packet.to_vec())
        .map_err(|e| e.to_string())?;
    shred.sanitize().map_err(|e| e.to_string())?;
    let root = shred.merkle_root().map_err(|e| e.to_string())?;
    if shred.signature().verify(leader.as_ref(), root.as_ref()) == false {
        return Err("leader signature does not authenticate Merkle root".into());
    }
    Ok(shred)
}

Shred::verify(leader) performs the variant-defined signed-data check and is preferred when supporting more than one format. The explicit form above makes the Merkle relationship visible.

Follow the protocol tree construction

Current Agave uses SHA-256 with domain-separated prefixes. Leaf hashing begins with byte 0x00 followed by SOLANA_MERKLE_SHREDS_LEAF. Internal hashing begins with 0x01 followed by SOLANA_MERKLE_SHREDS_NODE. Proof entries store 20-byte hash prefixes. Internal joining uses the 20-byte prefixes of both child hashes and produces a full 32-byte parent hash.

The local leaf index is the erasure-shard index: data position data_index - fec_set_index, or coding position k + coding_position. At each proof level, sibling order depends on the low bit of the current index, then the index shifts right. An odd last node is paired according to Agave's tree construction.

These details are precise but should remain in a compatibility adapter copied from the pinned release. A small difference in prefix, truncation, odd-node handling, leaf range, or proof offset creates a different root.

Derive proof size from the variant

The low four variant bits encode the number of proof entries. A standard 32-data plus 32-coding tree has six proof entries. Do not force six on every set. A shorter set can require a different proof size, and the parser must validate whether the claimed size fits the datagram and capacity rules.

The proof starts at a variant-dependent offset after the erasure material and chained-root region. Retransmitter-signed variants also reserve a signature region. Exact universal offsets are not currently specified by the product wire contract. Ask the typed shred for its root rather than slicing from an assumed tail position.

Enforce set consistency

All valid members of one FEC set must reconstruct the same root and carry the same leader signature. Cache the authenticated tuple (slot, fec_set_index, root, leader). Each new member still needs its proof checked against the root. Reject a member that reconstructs a different root even if its common key matches.

Coding configuration must also agree because local positions and tree size affect proof interpretation. A member cannot be moved between positions without changing the proof path.

After Reed-Solomon recovery, rebuild the entire tree from all reconstructed erasure members. Require its root to equal the authenticated root from received shreds. This validates recovered bytes and permits generation of proofs for missing members.

Validate chained roots

Chained variants include the preceding FEC set's root in the current set's committed material. For consecutive sets, require the current chained root to match the authenticated previous root. UDP can deliver sets out of order, so store a bounded pending link and validate when its neighbor arrives.

The first set's chain anchor and behavior across slot boundaries are release-defined. Use the pinned Agave logic. Do not invent an all-zero root rule. If the required predecessor has expired, report chain status as unresolved rather than treating an unchecked link as valid.

An optional retransmitter signature covers the Merkle root for the retransmission hop. Verify it only when the application has a trusted retransmitter identity mapping. The leader signature remains the authority for proposed ledger data.

Test adversarial changes

Flip one byte in the erasure leaf region, each proof entry, signature, common slot, data index, FEC-set index, coding position, and chained root. Every mutation that changes committed content must fail. Test a copied valid proof at another local index and members from two roots placed under one key.

Count parse failure, proof failure, leader-signature failure, root conflict, chain mismatch, chain unresolved, and retransmitter-signature result separately. These reasons lead to different operational responses.

Cache only authenticated roots

Key a root cache by cluster identity, slot, FEC-set index, root, and scheduled leader. Insert it only after a member proof and leader signature both pass. A root calculated from an unverified packet must never make later members cheaper. Expire cache entries with the same recent-slot horizon as duplicate and FEC state.

When a cached root exists, a new member still supplies an untrusted local index and proof. Sanitize its header, derive its erasure position, reconstruct its root, and compare the full 32 bytes. A 20-byte proof element is a protocol representation, not an acceptable root comparison length.

Measure proof calculation and Ed25519 verification separately. This identifies whether an upgrade changed tree work, signature batching, or leader-cache behavior. Keep cache hit rates and root conflicts visible, but never put root bytes in metric labels.