I've been working on RS2, a stream processing library that makes building real-time data pipelines in Rust much easier. It's designed for Kafka consumers, real-time analytics, media processing, and any scenario where you need to process high-throughput async streams efficiently.
This project contains over a year of programming. I tried to write this library with my best knowledge of Rust. Please let me know if there are parts I could improve
What makes RS2 special:
Rich Stream Combinators
Built-in Backpressure - Automatic flow control prevents memory explosions when producers are faster than consumers
Performance Metrics - Built-in observability with processing times, throughput, and error rates
Smart Queues - Bounded/unbounded async queues with proper cleanup and producer-consumer patterns
Parallel Processing - Easy CPU-bound and I/O-bound parallelization with automatic concurrency detection
Perfect for:
Kafka/message processing pipelines
Real-time analytics and aggregations
Media streaming and chunk processing
IoT sensor data processing
High-throughput API processing
The library handles all the async complexity, memory management, and performance optimizations so you can focus on your business logic.
Feedback welcome! Would love to hear thoughts from the Rust community on the API design and use cases.
Link: [crates.io: Rust Package Registry]

1 post - 1 participant

Read full topic