Production receiver checklist

Network and host

• Confirm the paid destination resolves to the intended public IPv4 address and UDP port.
• Permit UDP only from 64.130.40.90/32 at the edge and host firewall.
• Confirm the route, NAT mapping, and firewall remain stable through deployment and restart.
• Use an interface with at least 1 Gbps line rate and an effective MTU that carries a 1,228-byte UDP payload without fragmentation.
• Record the NIC driver, firmware, queue count, ring sizes, offload settings, kernel version, instance type, and CPU model.
• Synchronize the clock and export time offset. Use monotonic time for packet age.
• Reserve predictable CPU for receive and parsing. Measure steal, throttling, softirq, and run-queue delay.

Run a bounded packet capture during commissioning and verify destination port, source address, datagram length distribution, packet rate, and no IP fragmentation. Remove continuous capture from the hot path unless full retention is an explicit capacity-planned requirement.

Socket

• Set SO_RCVBUF before binding and read it back.
• Set net.core.rmem_max high enough for the requested value.
• Enable SO_RXQ_OVFL and parse its ancillary counter.
• Allocate at least 1,228 bytes for every receive slot and reject MSG_TRUNC.
• Batch receives with recvmmsg or prove one-call-per-packet has enough CPU headroom.
• Check the source IP again in the application.
• Keep the receive loop free of signature checks, FEC recovery, entry decoding, RPC, and synchronous logging.
• Use a bounded handoff queue with an explicit full policy.

Inspect the live socket with ss -u -l -n -m -p. Record the skmem receive limit and drop value. Inject receive-thread pauses up to the documented stall tolerance and require zero socket overflow at the capacity target.

Protocol validation

• Pin the Agave source or crate versions used for shred parsing and entry deserialization.
• Reject invalid lengths, variants, headers, versions, parent offsets, flags, sizes, coding positions, and erasure counts through the typed parser and sanitizer.
• Resolve the expected leader for each slot from a current schedule tied to the correct cluster and epoch.
• Verify the leader signature over the signed data defined by the shred variant.
• Verify each Merkle proof and enforce one authenticated root and compatible coding configuration per FEC set.
• Keep data and coding shred identifiers separate.
• Detect an identical duplicate and a conflicting payload for the same shred ID as different events.

Do not maintain private hard-coded offsets beyond a deliberately pinned, tested commissioning parser. Use the matching Agave implementation for variant-dependent proof, chained-root, retransmitter-signature, and data-capacity calculations.

Recovery and decoding

• Key recovery state by (slot, fec_set_index) and validate compatible headers before insertion.
• Read num_data_shreds, num_coding_shreds, and coding position from a sanitized coding header.
• Recover only when enough unique shards exist and data shards are missing.
• Recompute and validate the Merkle root after Reed-Solomon reconstruction.
• Order data shreds by slot and data index.
• Deshred only consecutive ranges ending at DATA_COMPLETE_SHRED.
• Deserialize entry payloads with the codec used by the pinned network release. Recent Agave uses wincode; older releases used bincode in this path.
• Extract VersionedTransaction values without pretending address lookup table indices are already resolved keys.
• Resolve v0 address lookup tables against state valid for the observed slot when full account keys are required.
• Treat shred observations as proposed data and reconcile against confirmed state later.

Test with missing data shards that can be recovered, loss beyond parity capacity, duplicate coding shreds, reordered arrival, conflicting headers, a wrong leader key, corrupted proof bytes, entry batches spanning FEC boundaries, empty ticks, legacy transactions, and v0 transactions with lookups.

Bounds and overload

• Bound every channel, packet pool, FEC map, slot map, duplicate cache, leader-key cache, lookup-table cache, and output queue.
• Apply time and capacity eviction. Prefer evicting whole expired protocol units over random individual packets.
• Export current count, byte estimate, oldest age, high-water mark, and eviction count for each state owner.
• Define whether overload preserves old completeness or current freshness.
• Reject downstream backpressure from the receive loop.

Replay a busy capture at one, two, and four times original timing. Require stable memory, no unaccounted loss, recovery after the burst, and queue age below the consumer deadline. Test on the deployed host rather than only a developer laptop.

Monitoring and alerts

• Collect NIC RX missed or no-buffer counters, softnet drops, UdpInErrors, UdpRcvbufErrors, UdpInCsumErrors, per-socket overflow, and application queue drops.
• Count received bytes and datagrams, parse outcomes, verification outcomes, duplicates, FEC recovery, unrecoverable sets, completed ranges, entry decode, transactions, and evictions.
• Measure age from receive to every major stage and export oldest queue age.
• Report required worker liveness and progress by partition.
• Keep metric labels bounded and sample detailed packet logs.
• Mark binary, configuration, kernel, and dependency deployments on dashboards.

Test every alert. Lower the socket buffer, pause a worker, block downstream output, inject invalid packets, stop feed delivery, and terminate a worker. Confirm the alert names the first known failing boundary.

Operations and release

• Start readiness only after socket options and workers are verified.
• Stop admission before draining workers during shutdown.
• Preserve startup configuration and last health snapshot in logs.
• Keep a rollback binary and its exact dependency lockfile.
• Replay the regression corpus before every protocol or compiler upgrade.
• Run a canary destination or shadow receiver when changing format code.
• Document the operator decision for upstream silence, kernel loss, protocol rejection spikes, memory pressure, and stale downstream output.

Sign off with evidence: a test timestamp, host identity, configuration hash, traffic sample, loss-counter deltas, latency percentiles, maximum queue ages, peak memory, and named reviewer. A checklist without captured results is a reminder, not a launch gate.