Hey everyone,

I’m creating qem, a text engine for Rust apps. It handles huge text files without loading everything into memory.

Qem isn't a GUI editor or a widget toolkit. It's an engine meant for tools like editors and log viewers. Fast access to large files is key.

Main features:

mmap reads for big files
incremental line indexing with a disk-side index
viewport-oriented reads instead of loading all
piece-table editing for large files
persistent session recovery with .qem.editlog
Document, DocumentSession, and EditorTab APIs for integration

Check out some recent numbers from stress tests on large files:

3GB file:

open: around 5ms
background index warm-up: about 3.1s
exact line count ready: around 5.3s
first edit at start: about 82ms
viewport after edit: about 310ms
64GB file:

open: about 447ms
background index warm-up: around 5.7s
first edit at start: about 74ms
viewport after edit: around 324ms
exact global line count still asynchronous at this size

One key detail: “exact total line count is still pending” doesn’t stop users from seeing the file. Visible lines open fast. Viewport reads still work, and the tail of the file is readable. Computing the exact total takes time, but interacting with visible parts is quick.

Qem focuses on huge file-backed workflows, unlike rope-based engines like Ropey, crop, or xi-rope-style approaches.

Those rope engines excel with in-memory text structures. Qem takes a different route for large files: it keeps clean files file-backed as long as possible, reads visible regions, and promotes to editable structures only when necessary.

So this isn’t “just another text rope.” It’s a practical engine for handling multi-gigabyte text files without full upfront materialization.

Links:

Crate: crates.io: Rust Package Registry
Repository: GitHub - mrhanty42/Qem: High-performance Rust text engine for massive files, with mmap-backed reads, incremental line indexing, and responsive editing. · GitHub
Releases: Releases · mrhanty42/Qem · GitHub
I welcome feedback from anyone working on editors, log viewers, or Rust tools for large text files.

1 post - 1 participant

Read full topic