Define receiver health checks

Separate liveness from readiness

Liveness answers whether the process can still supervise itself. Readiness answers whether it should receive or serve traffic. Progress answers whether shreds are moving through the pipeline. Do not collapse these into one Boolean.

Return liveness when the supervisor thread runs, the metrics endpoint responds, and no required worker has exited. Do not fail liveness because the UDP feed is quiet. Restarting a healthy process during an upstream or network interruption destroys useful diagnostic state and can repeat packet loss.

Return readiness only after the socket is bound, the effective SO_RCVBUF has been checked, all required workers are present, and the receiver can enqueue a local self-test object through its internal pipeline. A raw UDP destination normally has no load balancer that consumes readiness, but deployment tooling can use it to sequence startup and shutdown.

Add a feed-progress state

Track the monotonic time of the last accepted datagram from 64.130.40.90, last syntactically valid shred, last verified FEC set, last completed data range, and last decoded entry. These timestamps identify the stopped stage.

Do not set a hard promise about the maximum natural silence interval without measuring the network. A practical starting alert can mark feed progress degraded after two seconds and failed after ten seconds, but these thresholds are operational policy, not a Solana protocol guarantee. Correlate them with slot progression from an independent RPC source before paging.

Use a small state model:

use std::time::{Duration, Instant};

#[derive(Debug, PartialEq)]
enum State { Healthy, Degraded(&'static str), Failed(&'static str) }

struct Snapshot {
    now: Instant,
    last_packet: Instant,
    oldest_queue_age: Duration,
    live_workers: usize,
    required_workers: usize,
    recent_kernel_drops: u64,
}

fn assess(s: &Snapshot) -> State {
    if (s.live_workers == s.required_workers) == false { return State::Failed("worker count mismatch"); }
    if s.now.duration_since(s.last_packet) > Duration::from_secs(10) {
        return State::Failed("no accepted datagram for 10s");
    }
    if s.recent_kernel_drops > 0 { return State::Degraded("kernel UDP drops detected"); }
    if s.oldest_queue_age > Duration::from_millis(250) {
        return State::Degraded("queue age exceeds 250ms");
    }
    State::Healthy
}

fn main() {
    let now = Instant::now();
    let s = Snapshot { now, last_packet: now, oldest_queue_age: Duration::ZERO,
        live_workers: 4, required_workers: 4, recent_kernel_drops: 0 };
    if assess(&s) == State::Healthy { return; }
    std::process::exit(1);
}

Keep monotonic instants inside the process. Export seconds since last progress as a gauge. Wall-clock timestamps are useful for logs but can move when time synchronization adjusts the clock.

Check loss and saturation

Health should degrade when the delta of UdpRcvbufErrors, per-socket SO_RXQ_OVFL, NIC missed counters, softnet drops, or application queue discards is nonzero. A single packet loss may be repaired by FEC and may not justify a restart, but the receiver is not fully healthy during a loss interval.

Also check queue depth and oldest-item age. Depth alone is ambiguous because a large queue can be fresh during a burst. Age states the latency impact. Alert before the queue becomes full. For example, warn above 50 percent capacity for 30 seconds and fail service health when the oldest item exceeds its configured usefulness deadline.

Check memory use, open descriptors, CPU throttling, worker panic count, and buffer-pool availability. An empty pool means the next packet cannot be represented even if the socket remains readable.

Check protocol progress without inventing finality

Count valid headers, verified signatures, complete or recovered FEC sets, completed data ranges, decoded entries, and extracted transactions. A coding-only interval may advance FEC input without immediately producing transactions. Empty tick entries are valid. Therefore, transaction output alone is not a reliable feed health check.

Track the maximum observed slot and compare it to an independent, recent processed-slot RPC response. This comparison is diagnostic and should tolerate forks and timing differences. Shreds describe leader proposals, not confirmed chain state. Do not mark the feed corrupt because an observed slot is later absent from the confirmed chain.

Serve machine-readable details

Return a concise JSON object from an HTTP endpoint on a private management interface. Include overall state, reasons, receiver uptime, effective socket buffer, seconds since each progress timestamp, queue age and capacity, worker counts, and recent loss deltas. Do not include secrets or dump raw packet bytes.

Use HTTP 200 for live and degraded responses if orchestration would restart on a non-200 result. Provide a separate readiness endpoint that returns 503 when startup is incomplete or intentional shutdown has begun. Alerting should read the detailed state rather than relying only on HTTP status.

Test the checks

Stop the feed with a firewall rule in a staging environment, pause a worker, fill a bounded queue, lower SO_RCVBUF, inject malformed datagrams, and terminate one worker thread. Confirm each condition produces the expected reason and that removing it clears health without a process restart where appropriate.

Test the monitoring path itself. If metrics collection blocks the receiver, health tooling becomes a source of loss. Snapshot atomic counters and small immutable structures. Put expensive aggregation outside the ingest thread.

Keep a short rolling history of assessment inputs. An operator needs the counter delta and queue age that caused a transition, not only the current Boolean. Clear a degraded state after several clean intervals so a single good sample does not hide oscillation. Record every transition with the binary and configuration version.