Production receiver checklist

Network and host

• Confirm the paid destination resolves to the intended public IPv4 address and UDP port.
• At edge and host firewalls, restrict the registered destination port to UDP from 203.23.178.228/32.
• Confirm the route, NAT mapping, and firewall remain stable through deployment and restart.
• Use an interface with at least 1 Gbps line rate and enough effective MTU for 1,228 UDP payload bytes without fragmentation.
• Record the NIC driver, firmware, queue count, ring sizes, offload settings, kernel version, instance type, and CPU model.
• Synchronize the clock and export time offset. Use monotonic time for packet age.
• Reserve predictable CPU for receive and parsing. Measure steal, throttling, softirq, and run-queue delay.

Run a bounded packet capture during commissioning and verify destination port, source address, datagram length distribution, packet rate, and no IP fragmentation. Remove continuous capture from the hot path unless full retention is an explicit capacity-planned requirement.

Socket

• Set SO_RCVBUF before binding and read it back.
• Set net.core.rmem_max high enough for the requested value.
• Parse the ancillary overflow counter after enabling SO_RXQ_OVFL.
• Allocate at least 1,228 bytes for every receive slot and reject MSG_TRUNC.
• Batch receives with recvmmsg or prove one-call-per-packet has enough CPU headroom.
• Check the source IP again in the application.
• Keep the receive loop free of signature checks, FEC recovery, entry decoding, RPC, and synchronous logging.
• Use a bounded handoff queue with an explicit full policy.

Inspect the live socket with ss -u -l -n -m -p. Record the skmem receive limit and drop value. Inject receive-thread pauses up to the documented stall tolerance and require zero socket overflow at the capacity target.

Protocol validation

• Pin the Agave source or crate versions used for shred parsing and entry deserialization.
• Reject invalid lengths, variants, headers, versions, parent offsets, flags, sizes, coding positions, and erasure counts through the typed parser and sanitizer.
• Resolve the expected leader for each slot from a current schedule tied to the correct cluster and epoch.
• Verify the leader signature over the signed data defined by the shred variant.
• Verify each Merkle proof and enforce one authenticated root and compatible coding configuration per FEC set.
• Keep data and coding shred identifiers separate.
• Detect an identical duplicate and a conflicting payload for the same shred ID as different events.

Do not maintain private hard-coded offsets beyond a deliberately pinned, tested commissioning parser. Use the matching Agave implementation for variant-dependent proof, chained-root, retransmitter-signature, and data-capacity calculations.

Recovery and decoding

• Own recovery state under (slot, fec_set_index), validating header compatibility before insertion.
• From the sanitized coding header, read coding position together with num_data_shreds and num_coding_shreds.
• Recover only when enough unique shards exist and data shards are missing.
• After Reed-Solomon reconstruction, recompute its Merkle root and validate the result.
• Order data shreds by slot and data index.
• Require a consecutive range whose final member carries DATA_COMPLETE_SHRED before deshredding.
• Deserialize entry payloads with the codec used by the pinned network release. Recent Agave uses wincode; older releases used bincode in this path.
• Extract VersionedTransaction values without pretending address lookup table indices are already resolved keys.
• When full account keys are required, resolve v0 lookups through address tables valid at the observed slot.
• Keep shred observations as proposed data, then reconcile them with later confirmed state.

Test with missing data shards that can be recovered, loss beyond parity capacity, duplicate coding shreds, reordered arrival, conflicting headers, a wrong leader key, corrupted proof bytes, entry batches spanning FEC boundaries, empty ticks, legacy transactions, and v0 transactions with lookups.

Bounds and overload

• Bound every channel, packet pool, FEC map, slot map, duplicate cache, leader-key cache, lookup-table cache, and output queue.
• Apply time and capacity eviction. Prefer evicting whole expired protocol units over random individual packets.
• Export current count, byte estimate, oldest age, high-water mark, and eviction count for each state owner.
• Define whether overload preserves old completeness or current freshness.
• Reject downstream backpressure from the receive loop.

Replay a busy capture at original timing, then at twice and four times that speed. Require stable memory, no unaccounted loss, recovery after the burst, and queue age below the consumer deadline. Test on the deployed host rather than only a developer laptop.

Monitoring and alerts

• Collect NIC counters for missed RX or absent buffers, softnet drops, UdpInErrors, UdpRcvbufErrors, UdpInCsumErrors, per-socket overflow, and application queue drops.
• Count received bytes and datagrams, parse outcomes, verification outcomes, duplicates, FEC recovery, unrecoverable sets, completed ranges, entry decode, transactions, and evictions.
• Measure age from receive to every major stage and export oldest queue age.
• Report required worker liveness and progress by partition.
• Keep metric labels bounded and sample detailed packet logs.
• Mark binary, configuration, kernel, and dependency deployments on dashboards.

Test every alert. Lower the socket buffer, pause a worker, block downstream output, inject invalid packets, stop feed delivery, and terminate a worker. Confirm the alert names the first known failing boundary.

Operations and release

• Start readiness only after socket options and workers are verified.
• Stop admission before draining workers during shutdown.
• Preserve startup configuration and last health snapshot in logs.
• Keep a rollback binary and its exact dependency lockfile.
• Replay the regression corpus before every protocol or compiler upgrade.
• Run a canary destination or shadow receiver when changing format code.
• Document the operator decision for upstream silence, kernel loss, protocol rejection spikes, memory pressure, and stale downstream output.

Sign off with evidence: a test timestamp, host identity, configuration hash, traffic sample, loss-counter deltas, latency percentiles, maximum queue ages, peak memory, and named reviewer. A checklist without captured results is a reminder, not a launch gate.