My anchored-leveldb project is now fully capable of reading LevelDB databases.

Source code: GitHub - robofinch/anchored-leveldb at alpha · GitHub

Discussion on Reddit: Reddit - Please wait for verification

Context

LevelDB is a popular database format created by Google which is used in Google Chrome, Minecraft: Bedrock, and Bitcoin, among many other projects. Nevertheless, Google has not spent much effort on improvements or even bug fixes for LevelDB in the past years. While working on a project that involves reading Minecraft: Bedrock worlds' LevelDB databases, I found that I was bottlenecked by rusty-leveldb (a thread dedicated to database iteration could not feed my other worker threads quickly enough). After looking through the rusty-leveldb and google/leveldb codebases, I set out to make a better LevelDB implementation than what was available.

Project status

anchored-leveldb patches multiple bugs in Google's leveldb library^[1] as well as Mojang's fork^[2], while achieving ~16% better sequential read performance under low memory pressure. Under high memory pressure, anchored-leveldb is substantially (~33%) faster than the others.

However, this release is anchored-leveldb v0.0.1-alpha; my implementation is not yet capable of writing LevelDB databases (though the groundwork is there). The documentation is lacking, error enums are messy, setting the the database config is not as ergonomic as I want^[3], some unsafe code is still undocumented... I had to rush this out the door for an undergraduate senior thesis. The thesis analyzes LevelDB codebases to find possible improvements in performance and correctness, as well as providing this proof-of-concept implementing some of those improvements. I was shocked to see that my code is already faster than others, when there's still more improvements planned.

If you decide to use anchored-leveldb v0.0.1-alpha to read LevelDB databases, I'd recommend operating on a copy of the database, in case there are uncaught bugs capable of corrupting the database.

Plans for beta

In a few months, I'll have a fully-featured (and fully-tested) beta release. The intent of this post is more-or-less to put this project on your radar.

When I release the beta, I'll also provide simple programs that scan through LevelDB databases or Minecraft: Bedrock worlds in particular, for the sake of finding possible flaws in this implementation and, critically, finding real-world examples of corrupted LevelDB databases or Minecraft worlds (which will inform the creation of tools for recovering corrupted data). For any Minecraft: Bedrock players (or anyone else who happens to have small or medium LevelDB databases lying around), I'd greatly appreciate any help in gathering data at that time.

1. This includes use-after-free and buffer overrun bugs, as well as logic bugs capable of corrupting databases. ↩︎

2. Mojang added the ability to "suspend"/pause compactions in a very bolted-on manner which interacts poorly with other parts of the leveldb codebase. I've elevated this to a full-fledged feature for pausing and resuming compactions. ↩︎

3. I'm proud to say that anchored-leveldb is much more configurable than other LevelDB implementations. ↩︎

1 post - 1 participant

Read full topic