Hi fellow Rustaceans!

I recently built a VS Code extension to solve a personal pain point: opening multi-gigabyte log files without freezing the editor (which usually happens due to Electron/DOM memory limits).

I decided to offload all the heavy lifting to a Rust sidecar binary. I’m posting this here because I’d love some feedback on the backend implementation and architecture.

The Architecture

The extension uses a standard stdio JSON IPC pattern.

1. Frontend: VS Code Webview (TypeScript) handles virtual scrolling and rendering.

2. Backend: A Rust binary (log-core) manages the file I/O and searching.

Technical Highlights (The Rust part)

The backend is built around memmap2. Here is how I approached the performance bottlenecks:

• Memory Mapping: I map the entire file into the virtual address space. This allows the OS to handle paging. Opening a 10GB file results in negligible heap usage for the process.

• Indexing: On file open, I perform a single linear scan O(n) to identify line breaks (\n). I store line offsets in a Vec.

  • Performance: This creates a lightweight index that allows random access to any line number instantly.

• Reading: When the UI requests lines 500-600, I slice the byte range from the mmap using the index and convert it using String::from_utf8_lossy (to handle potential binary garbage in logs without panicking).

• Search: I use the regex crate. To avoid blocking the IPC thread on massive files, the search is chunked, and results are streamed back to the frontend.

• Cross-Compilation: I set up GitHub Actions to compile targets for x86_64-unknown-linux-gnu, x86_64-pc-windows-msvc, x86_64-apple-darwin and aarch64-apple-darwin.

Why Rust?

I needed something with zero startup time (unlike JVM) and safe memory management for string slicing. The ability to easily cross-compile a single static binary for all major platforms was the deciding factor versus C++.

Code & Feedback

The project is open source. I’m particularly interested in feedback regarding:

1. Safety around the mmap implementation (handling external file truncation while mapped).

2. Optimizations for the indexer (currently using a simple iterator).

Repository
Link to Marketplace

Thanks for checking it out!

1 post - 1 participant

Read full topic