Combined update on two safe-Rust compression crates.

lz4rip (repo)

v0.1.0 => v0.8.5 since the initial post:

• Workspace split into lz4rip-encode / lz4rip-decode / lz4rip-core. Separate LLVM compilation units eliminate a class of regressions where dead encoder code shifts register allocation in the decoder. Decompress improved 4-17% across Silesia; gap vs C lz4 narrowed from 10-33% to 2-21%
• 8 KB hash tables, half the L1d footprint of C lz4 and lz4_flex. Epoch-based table reuse skips clearing between calls for small inputs. Dual-table dictionary compression uses a read-only pristine table probed on main-table miss.
• no_std / no-alloc: block compress and decompress work without std or alloc. Hash tables go on the stack (~8 KB).
• Built-in dictionary training: DictTrainer (COVER-based) learns a dictionary from your data. No external tools needed.

x86_64 summary chart | aarch64 summary chart

zrip (repo)

v0.1.0 => v0.3.4 since the initial post:

• Workspace split into zrip-encode / zrip-decode / zrip-core.
• Encode +4-17%, decode +5-24% across all levels via instruction count reduction, branch hints, and inlined rep offsets.
• Streaming dictionary compression: FrameEncoder::with_dict(), FrameDecoder::with_dict(), CompressContext::with_dict() for buffer-reusing streaming with trained dictionaries.

Fastest pure-Rust zstd encoder that isn't just unsafe from beginning to end. libzstd-rs-sys is a c2rust translation of C zstd with pervasive unsafe throughout, and structured-zstd is similarly unsafe-heavy. zrip is safe Rust you can actually audit.

x86_64 summary chart | aarch64 summary chart
x86_64 scatter chart | aarch64 scatter chart

All algorithm logic in both crates enforces #[forbid(unsafe_code)]. Unsafe is confined to raw memory operations whose bounds are proven by the surrounding safe code.

lz4rip is used in OMQ.rs, a Rust implementation of ZMQ, for message-level compression in tight loops where compress+send+decompress latency dominates. The results are nuts, especially with a trained dictionary.

1 post - 1 participant

Read full topic