Hi folks,

I'm trying to deserialise a very large JSON (from a single String) into a struct with hundreds of nested structs in an async program.

I'm still working on the struct by adding more and more structs and hit an issue with stack overflow.

I added some Boxes like this:

pub struct NestedStructWithOtherNestedStructs {
    pub a: Option<Box<Foo>>,
    pub b: Box<Bar>,
}

And that helped for some time, but then again, I started getting the stack overflow.

Could you please describe your strategy for something like this?

I tried to write a trait that works as a wrapper for deserialization into a Box, but that didn't help.

impl ...
    fn my_deser(inp: String) -> Result<Box<Self>, Error>
    where
        Self: serde::de::DeserializeOwned,
    {
        Ok(serde_json::from_slice(out)?)
    }

I experimented with running the deserialization in another thread:

tokio::task::spawn_blocking(move || ...).await

And that worked but I had to start Tokio with a large .thread_stack_size(32 * 1024 * 1024).

I'm not trying to save memory in this program, but I think this is wrong since I'm bumping the stack for all tokio threads just because of this single deserialization.

Any ideas welcomed ;-). Thank you!

3 posts - 2 participants

Read full topic