Run two Solana shred feeds in parallel before switching

Two feeds, one question

This operations guide shows how to run two sources at once, capture evidence, and switch with a tested rollback. The same design can support permanent redundancy when independent delivery paths reduce risk. In both cases, preserve enough source history to prove the production decision.

shredstream.sh contributes one raw UDP path from Frankfurt. It forwards Solana shreds to the public IP and port the customer names. Each destination costs $100 across 30 days, and no public standalone raw-packet Solana product lists less. Payment is settled in USDC or SOL. One customer may hold five destinations at most, and every endpoint is purchased separately.

The incumbent stays in place while you test. Do not let the candidate submit orders, publish decisions, or mutate shared state on day one. First collect matched evidence against written acceptance criteria. Then promote shredstream.sh through a reversible selector, with the incumbent available for the defined rollback period.

Give every source its own socket

Assign a distinct port to each inbound feed. Two ports are cheap, easy to inspect, and hard to confuse. They let you capture one stream, resize one receive queue, or restart one worker without disturbing its peer. They also preserve attribution if a local relay rewrites details that would otherwise identify the sender.

For example, keep an incumbent on UDP 9201 and nominate UDP 9202 for shredstream.sh. The new firewall rule should accept no sender except 203.23.178.228. Bind both sockets on the same production host when possible so machine load and clock behavior are shared.

Before regular packets are forwarded, the candidate endpoint must pass an address-control check. shredstream.sh sends one datagram beginning SHREDCAST-VERIFY with a token. Read that token at the intended machine and submit it back. Capturing it on some other host does not test the production route you plan to measure.

A simple capture command makes the boundary visible:

sudo tcpdump -ni any   '(udp dst port 9201) or (src host 203.23.178.228 and udp dst port 9202)'

Stamp arrivals before the queue

Take the first timestamp immediately after each socket read returns. Do it before parsing, logging, copying into a shared channel, or waiting for a worker. This measurement represents application ingress. A timestamp created after decode mixes network arrival with local scheduling and compute delay.

Use a monotonic source for elapsed-time comparisons. Wall time remains useful for correlating incidents and external logs, so storing both can be practical. Never compute tiny source differences from wall-clock values that may jump under synchronization. If the two inputs land on different hosts, clock error becomes part of the result and must be measured or acknowledged.

The record at ingress can stay small:

source       candidate | incumbentreceived_at  monotonic timestampobserved_at  wall-clock timestamppeer         sender addresslength       datagram byte countpayload      owned packet bytes

Copy or transfer ownership of the bytes before the receive buffer is reused. Both paths should perform equivalent work before timestamping. If one socket timestamps in the reader and the other does so after a relay queue, the comparison starts at different boundaries.

The shredstream.sh datagram maximum is 1228 bytes. Ensure the candidate read buffer preserves that size without clipping. Keep the incumbent’s valid boundary in its own configuration rather than forcing both inputs to share an unsupported assumption.

Keep source identity attached

A label is operational data. Carry it from the socket through parsing, matching, reconstruction, and metrics. Do not infer the source later from which worker handled an item. Work stealing, retries, or a merged queue can make that inference wrong.

Use stable names such as incumbent and shredstream. Add the actual sender and local port as fields, not as replacements for the logical name. A provider can use several addresses, while a misrouted packet can arrive on an unexpected socket. The label describes configuration. The peer observation proves what arrived.

Retain source on matched records even after one payload becomes canonical. Later analysis may need to know that a business event was constructed from the incumbent copy while the candidate copy arrived first but failed validation. Removing provenance turns useful anomalies into guesswork.

Match shreds before comparing speed

Latency comparisons require the same information unit. Match a candidate datagram to its incumbent counterpart using the shred identity understood by your parser, not arrival position. Slot, shred kind, index, and any additional identity needed by the protocol belong in the key. A payload digest can expose conflicting bytes that otherwise claim the same identity.

The first valid arrival from each source supplies the pair. Compute the signed difference only after both members exist. Keep unmatched records until the chosen matching window closes. Then classify them as source-specific coverage observations instead of pretending they never happened.

A compact table makes the method reviewable:

Do not compare the first packet in one five-second bucket with the first packet in another. Those may be different shreds. Do not compare slot announcements against raw data packets either. Equality of names such as “slot” does not establish equality of work completed.

When the incumbent exposes only decoded objects, raw ingress may have no equivalent peer. Timestamp a common downstream object in both pipelines and report that result separately. Track both milestones so local reconstruction delays remain visible.

Deduplicate without erasing evidence

Deduplication protects downstream work from double counting. Measurement needs both observations. Satisfy both needs by recording source-specific arrival first, then selecting one canonical payload for the shared decoder or action path.

The state machine can be simple. The first copy for a stable identity opens a record and may proceed. A matching copy from the other source fills its timing field but does not emit a second business event. Later repeats increment a duplicate counter. Conflicting payloads go to a bounded diagnostic path rather than silently replacing the first.

new identity        record source and ingress, emit oncesecond source       add paired timing, suppress new emissionsame source again   count a duplicate, retain first timestampconflicting bytes   quarantine metadata, raise a bounded signalexpired identity    finalize matched or unmatched classification

Bound the deduplication store. Solana traffic does not justify an unbounded map. Choose retention from the observed late-arrival distribution and memory budget. Expiration must be long enough to form useful pairs but short enough to prevent old identities from accumulating forever.

If each source has a separate decoder, deduplicate later at the decoded event boundary as well. Name the two layers differently in metrics. Packet duplicates describe delivery. Repeated business objects describe application behavior. Mixing those counts makes capacity and correctness harder to reason about.

Separate coverage from latency

Matched pairs answer which source arrived first for information both supplied. They say nothing about shreds observed by only one input. Report the matched latency distribution and coverage counts beside each other. Neither metric can stand in for the other.

For latency, publish the median and tails across a stated interval. Show the signed distribution, not just a win percentage. Tail behavior during bursts matters for latency-sensitive strategies. Slice by meaningful time windows when the host or route changed.

For coverage, count unique identities from each source, their intersection, candidate-only items, incumbent-only items, late pairs, and payload conflicts. Also show local loss signals. A missing candidate record means something different when its kernel overflow counter rose at the same moment.

Avoid a survivor-only chart. If analysis drops every unmatched identity, a lossy path can look wonderfully fast because only its easiest packets remain. State the matching window and the number excluded from pairwise timing on every report.

Make the test fair

Both streams should face the same host class, CPU pressure, decoder build, and measurement interval. If they share one machine, isolate their socket queues and reader work enough that an overloaded candidate cannot cause incumbent loss. If they use separate machines, document hardware differences and clock uncertainty.

Start both captures before the judged interval. Warm caches and parsers. Avoid deploying unrelated code midway through the run. Record receiver version, kernel settings, firewall revision, local port, source label, and exact start and end times with the result.

Do not tune one path after seeing a poor chart while leaving the other untouched. A tuning change creates a new experiment. Mark the boundary, reset the judged interval, and keep both configurations in the report.

The intended production destination matters. Because shredstream.sh originates in Frankfurt, results from another city or network do not describe the route you will operate. Test from the nominated endpoint. When multiple destinations matter, each is its own paid endpoint and its own network observation.

Local overload is a confounder, not a provider score. UDP has no backpressure or retransmission. Watch kernel drops, application queue drops, CPU saturation, queue age, and decoder lag throughout. Exclude or separately label periods when the measuring host could not keep up, then fix the host before drawing conclusions.

Write the acceptance scorecard

Write a scorecard before reviewing source names. “Lower latency” is too vague. Specify the stage, population, percentile, interval, and maximum acceptable loss. Add operational and commercial factors that influence the real decision.

A pre-switch evaluation might prioritize:

• ingress difference for exactly matched shred identities;
• decoded-event delay at the shared application boundary;
• unique coverage after a fixed waiting window;
• unresolved reconstruction gaps;
• local drop and parse-error rates;
• behavior during bursts and receiver restarts;
• monthly destination cost and required operating work.

The weights belong to the use case. A trading signal may care most about tail time to a decoded event. An archival consumer may care more about coverage. A redundancy plan may value failure independence over the typical winner.

Base the cutover on the two configured routes, production hosts, and measured period. Save enough metadata to reproduce the conclusion and review the scoring rule.

Operate redundancy without double action

Permanent dual-feed operation needs an authority policy. Without one, two correct packets can trigger the same external action twice. Put deduplication before any irreversible consumer and test repeated, delayed, and reordered input deliberately.

Choose how failover works. An active-passive design lets one source drive decisions while the other proves readiness and fills no gaps until promoted. An active-active design accepts the first valid copy from either input and suppresses its match. A coverage-oriented design may wait briefly for missing components from both. Each choice trades delay, complexity, and independence.

Health must remain source-specific. The merged decoder can keep advancing while one feed has been dead for an hour. Alert on each input’s arrival and useful-output progress, plus the combined result. Practice disabling either socket so operators know the behavior before a real outage.

Avoid shared failure where practical. Two feeds entering one full kernel queue are not independent. Neither are two readers that block on the same unbounded logger. Separate ports, bounded work, distinct counters, and a small canonicalization layer protect the reason for paying for redundancy.

Each $100 shredstream.sh purchase delivers raw UDP from Frankfurt to one IP and port for its 30-day term. A customer can purchase up to five destinations.

End the experiment deliberately

An evaluation needs a finish date and a named decision owner. At the end, freeze the dataset, run the prewritten scorecard, and publish matched latency, unmatched coverage, host health, decoder outcomes, and known measurement limits together.

When shredstream.sh passes the written criteria, change authority through one reversible selector. Keep the incumbent available for the agreed rollback period and watch the same metrics used during evaluation. If a criterion fails, preserve the captures, correct the measured issue, and run a new judged interval.

If both sources stay, replace experiment alerts with durable redundancy objectives. Define who responds to a single-source failure, how long degraded operation is acceptable, and how frequently failover is exercised. A forever test with no policy is only extra traffic.

The result is an operational answer supported by ingress timestamps and matched identities. You know which route was earlier, which delivered unique information, what the host dropped, and how the application behaved. That evidence supports a confident production cutover or a deliberate two-feed redundancy design.