I'm currently using Tokio to handle 2 main tasks in my architecture:

• A file watcher, which monitors changes to files in a folder (using the notify crate).
• An event processor, which consumes the deserialized events from the file watcher and updates its internal state.

I'm having issues with getting the correct setup, and constantly run into issues:

• Using std::fs::File gives me high read speed on the updated file, but because files are updated at a high frequency, the file watcher tends to starve the event processor (from tokio-console, the event processor is busy 21s, scheduled 1m08s and idle 2m40s).
• Using tokio::fs::File lowers my reading speed, I can see that I'm getting less throughput on the event processor, but the scheduling looks a lot healthier (from tokio-console, the event processor is busy 34s, scheduled 27s and idle 6m22s).

I'm wondering if there's a way to keep sufficient throughput, all the while maintaining healthy scheduling for my tasks? Currently I've been considering adding a tokio::task::yield_now in the file watcher to periodically yield to the event processor every x events processed.

1 post - 1 participant

Read full topic