Deduplicate shreds

Define identity correctly

Use (slot, index, shred_type) as the shred ID. Data and coding indices occupy separate spaces. Do not include FEC-set index in identity because a conflicting packet can claim a different FEC set while occupying the same slot, index, and type. That disagreement is evidence, not a second legitimate identity.

Use (slot, fec_set_index) separately for recovery ownership.

Separate repeats from conflicts

An identical authenticated shred received twice is a duplicate. The second copy does not add a Reed-Solomon position and should not repeat signature verification, FEC insertion, or decoding if the first accepted result remains cached.

The same shred ID with different leader-authenticated content is a conflict. Do not let the later packet overwrite the earlier one. Count it, preserve bounded evidence, and quarantine according to protocol policy. A bit error, mixed feed, malicious packet, or leader equivocation can produce this condition.

Retransmitter-signed Merkle variants can differ only in the trailing retransmitter signature while retaining the same leader-authenticated shred. Current Agave's Shred::is_shred_duplicate comparison ignores the format-defined retransmitter signature region when deciding whether same-ID payloads conflict. Use it instead of trimming a guessed number of bytes from every packet.

use solana_ledger::shred::Shred;
use std::collections::hash_map::{Entry, HashMap};

#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
struct Id { slot: u64, index: u32, code: bool }

enum Admit { New, Repeat, Conflict }

fn admit(cache: &mut HashMap<Id, Shred>, incoming: Shred) -> Admit {
    let id = Id { slot: incoming.slot(), index: incoming.index(), code: incoming.is_code() };
    match cache.entry(id) {
        Entry::Vacant(v) => { v.insert(incoming); Admit::New }
        Entry::Occupied(o) if o.get().is_shred_duplicate(&incoming) => Admit::Conflict,
        Entry::Occupied(_) => Admit::Repeat,
    }
}

The method name follows Agave terminology: it reports a duplicate-shred conflict, meaning the same ID has different underlying payload. Equal underlying content returns false and is a repeat in this classifier.

Place deduplication in two stages

Before cryptographic work, use a short-lived fingerprint cache only for exact bytes already authenticated. Key it with a fast digest plus length, and confirm equality before skipping work. Never treat a digest collision as equality. This stage reduces repeated proof and signature cost.

After parsing, use the full shred ID and canonical protocol comparison to find conflicts. Store the first authenticated typed shred or canonical digest until the slot horizon expires. A packet that has not authenticated cannot replace or conflict with trusted state, but its rejection should still be counted.

Multiple delivery providers or Turbine paths can create legitimate repeats. Source address is delivery metadata, not part of shred identity. Record a bounded source bitset or count if path comparison matters, but do not process the same shred once per source.

Choose a cache horizon

Retain IDs for active and recently completed slots long enough to cover UDP reordering and late coding shreds. Bound by maximum slots, entries, bytes, and age. Removing a completed FEC set from recovery does not require removing its duplicate keys immediately. A late repeat should remain cheap.

At the measured 5,585 packets per second, a ten-second full packet cache receives about 55,850 packet events before duplicates. Storing complete 1,228-byte buffers would consume about 69 MB before map overhead. Store a strong digest and compact metadata when full bytes are no longer needed for recovery or conflict evidence.

For conflicts, retain both canonical digests, selected header fields, arrival times, source metadata, verification result, and a bounded raw sample. Do not permit repeated conflicts to allocate without limit.

Handle races

Partition by shred ID or perform an atomic map entry operation. Two workers that both observe absence can otherwise verify and insert the same packet twice. FEC ownership by set does not fully solve duplicate races if verification occurs in a general pool.

Keep the state transition explicit: unseen, verifying, accepted, rejected, conflict, expired. A failed verification must not poison the ID so a later valid packet is dropped. A concurrent valid repeat can wait for the first result or verify independently under a bounded policy.

Do not deduplicate transactions at this layer

Shred deduplication and transaction deduplication answer different questions. The same transaction signature can appear in competing proposed slots or be retried after a fork. Preserve its slot and entry context. Decide downstream whether repeated transaction observations are collapsed, reconciled, or emitted as lifecycle events.

Test canonical behavior

Test identical bytes, a different retransmitter signature, one changed leader signature, changed proof byte, changed FEC-set index, data and coding with the same numeric index, two concurrent copies, cache expiry, and a rejected packet followed by a valid packet. Require exactly one FEC admission for repeats and a distinct conflict event for changed authenticated content.

Export repeats saved before verification, repeats after parsing, conflicts, cache entries, cache bytes, evictions, and race waits. A high repeat rate is not loss, but it consumes receive and lookup capacity.

Include the cache generation in restart diagnostics. A process restart naturally forgets recent identities and can raise repeat work without changing the feed.