Monitor your receiver

Measure rates, loss, latency, and state

Expose counters for events, gauges for current state, and histograms for duration or size. Keep labels bounded. Slot, FEC-set index, shred index, transaction signature, source port, and error text are high-cardinality values and do not belong in metric labels. Put sampled examples in structured logs.

Start at the socket with receiver_datagrams_total, receiver_bytes_total, receiver_wrong_source_total, receiver_truncated_total, and receiver_socket_overflow_total. Export the effective SO_RCVBUF as receiver_socket_buffer_bytes. Collect host UdpInErrors, UdpRcvbufErrors, UdpInCsumErrors, softnet drops, and the deployed NIC's missed or no-buffer counters.

At parsing, count shred_parse_total{result}, with a small result set such as ok, invalid_length, invalid_variant, invalid_header, and unsupported_version. At verification, count shred_verify_total{result}, separating missing leader key from invalid signature.

At FEC, export active sets, stored bytes, duplicate shards, recovered data shards, unrecoverable sets, conflicting headers, and expired sets. At reassembly, count completed data ranges and missing-index failures. At decode, count entry batches, entries, transactions, codec errors, and address-lookup requirements.

Instrument the hot path cheaply

Use relaxed atomic counters for independent event totals. Snapshot them from a metrics thread. Histograms can be thread-local and merged periodically. Do not take a global map lock for every packet.

use std::sync::atomic::{AtomicU64, Ordering};

#[derive(Default)]
struct Metrics {
    datagrams: AtomicU64,
    bytes: AtomicU64,
    queue_drops: AtomicU64,
    parse_errors: AtomicU64,
}

impl Metrics {
    fn received(&self, len: usize) {
        self.datagrams.fetch_add(1, Ordering::Relaxed);
        self.bytes.fetch_add(len as u64, Ordering::Relaxed);
    }
    fn snapshot(&self) -> [u64; 4] {
        [self.datagrams.load(Ordering::Relaxed),
         self.bytes.load(Ordering::Relaxed),
         self.queue_drops.load(Ordering::Relaxed),
         self.parse_errors.load(Ordering::Relaxed)]
    }
}

fn main() {
    let metrics = Metrics::default();
    metrics.received(1216);
    let _snapshot = metrics.snapshot();
}

For durations, record receive-to-parse, receive-to-verified, receive-to-FEC-complete, receive-to-entry, and receive-to-consumer. The arrival Instant must travel with the packet or completed unit. Stage execution time alone misses queue wait, which is often the main latency under load.

Build a useful dashboard

Put input packet rate and Mbps first. Compare them with the measured reference of 5,585 packets per second and 54.3 Mbps, but do not alert on a fixed deviation alone because Solana production varies. Beside input, graph host and application drops as per-second deltas.

Show each queue's depth, capacity, oldest-item age, enqueue failures, and worker throughput. A pipeline view should make the first growing queue visible. Add CPU by thread or process, softirq CPU, memory, buffer-pool free slots, and scheduler steal or throttling.

Show protocol outcomes: valid shred rate by data or coding type, duplicate rate, signature failure rate, active FEC sets, recovered shard rate, unrecoverable set rate, completed data ranges, entries, and transactions. Graph maximum observed slot and the gap to an independent processed slot as a diagnostic.

Alert on symptoms with causes

Page immediately on any sustained UdpRcvbufErrors, per-socket overflow, NIC missed packet counter, or application ingest queue drop. Warn on transient single increments and page when loss continues across two or more sampling intervals. The exact policy depends on whether FEC recovery preserved all data.

Alert on oldest queue age before capacity reaches full. Alert when any required worker makes no progress while its input queue is nonempty. Alert on signature failures above a small baseline, but distinguish a leader-key lookup failure from cryptographic rejection. A missing or stale leader schedule is an operational dependency failure.

Alert on memory state, not only resident size. Active FEC sets and retained bytes should remain within calculated bounds. A monotonically growing duplicate cache or slot map is a leak even if the host has not reached memory pressure.

Log evidence without flooding

Emit a structured startup record containing bind address, source allowlist, effective receive buffer, batch size, queue capacities, worker counts, binary version, and pinned Agave revision. Log state transitions and sampled failures. Include slot, shred type, index, FEC-set index, datagram length, and reason where parsing got far enough.

Rate-limit repeated errors by reason. A malformed packet storm must not saturate disk or block stdout. Keep total counters exact even when logs are sampled. Never log full transaction payloads by default because they add volume and may expose data the consumer did not intend to retain.

Reconcile counters

Over an interval, accepted datagrams should equal parsed successes plus parse rejects plus pre-parse application discards, allowing for in-flight queue change. Verified inputs should reconcile with verification failures, duplicates, and FEC admissions. Completed ranges should reconcile with decode successes and failures.

Write these invariants into tests and dashboard recording rules. A monitoring counter that cannot be reconciled is often incremented at the wrong boundary. Reset-aware rate calculations must handle process and host restart.

Retain enough history to compare kernel, dependency, and deployment changes. Mark releases and configuration changes on charts. Use captured traffic replay for performance regression tests because synthetic fixed-size UDP packets do not exercise the protocol state or allocation pattern.

Define metric ownership before launch. The host agent owns NIC, softnet, and UDP MIB counters. The receiver owns its socket overflow, queues, protocol outcomes, and end-to-end age. Give every alert one runbook link and one first diagnostic query. Review unused alerts and missing counters after each incident.