A practical follow-up to my earlier TRAINS post. TRAINS is a formally-verified total-order broadcast primitive; trains-valkey is the payoff — and it is probably the part most people actually care about.

The problem: Redis/Valkey replication is asynchronous, so a failover can drop acknowledged writes. Sentinel trades durability for availability — if the primary dies after acking a write but before it replicates, that write is gone.

What it is: a proxy (Rust) that puts an unmodified Valkey/Redis behind the TRAINS ring. Writes are totally ordered and replicated across the ring before they are acknowledged to the client, so a node can be SIGKILLed mid-flight without losing an acked write. Reads bypass the ring.

The Rust-relevant parts:

• async proxy speaking RESP, with an unmodified valkey underneath — no fork, no patch to the server
• rustls for the ring transport (no OpenSSL)
• the durability guarantee — no acknowledged-write loss across a crash — falls out of the ring total-order + pre-ack replication, which is the property I checked seven ways in the parent project
• chaos-tested on EC2: N-node clusters, random SIGKILL of the primary under load, asserting no acked write is lost and survivors converge byte-identically

Honest about limits: it is consistency-first and control-plane-scale — small messages, small clusters. It halts under partition by design (PC/EC in PACELC), so it is not a drop-in for a high-throughput data plane. The point is loss-free failover for the cases where losing a write is worse than briefly pausing.

Code (MIT): GitHub - yeychenne/trains-valkey: Loss-free Valkey/Redis failover via TRAINS total-order broadcast (state-machine replication) · GitHub
Write-up: https://yeychenne.hashnode.dev/trains-valkey-no-write-loss

Happy to go into the proxy architecture, the RESP handling, or how the durability property maps onto the ring.

2 posts - 1 participant

Read full topic