I am writing a library (using nightly rust). Its high-level interface is async, with 2 threadpools for various calculation-intensive tasks. It is supposed to be used on servers (with HTTP interface) and in downloadable applications (rust async interface).

One of tasks iran in a threadpool is a data processing task. When a user makes request to do something and provides previously unseen data, a cache generation process begins. It reads data, and extracts small bits necessary for the lib to function. This process can run from time to time, but is not expected to run often.

Depending on implementation, it spikes memory use of the process (http interface to the lib) to 600-1300 MB. Outside of it, process does not need more than 50-200 MB.

The issue with it, is that this memory is not returned to the system by the default allocator. After the cached data generation is complete, memory use stays at 1G+. I tried calling malloc_trim after it, and it helps when it comes to memory. However, info everywere suggests I shouldn't be doing that in a lib, so I am looking for different solutions.

One of ideas is to have generation-involved data allocated in a separate memory region, and then free it up as soon as the generation is complete. Is it something doable in rust? Do I need to code my own allocator for that? Or can I use an "instance" of some already coded allocator, with its lifetime being tied to the data generation process (data generation is done - memory is released back to the OS)?

8 posts - 5 participants

Read full topic