I'm trying to troubleshoot an application that seems to be leaking some async tasks.

I'm using tokio::runtime::RuntimeMetrics and I'm monitoring alive_tasks.
I'm testing my application by sending some load, then going idle for like 30s, in an endless test loop. Over time, I see how alive_tasks keeps growing.

After trying to add logs and also ensure JoinHandles are at least aborted on drop, I still see alive_tasks growing (and base memory usage slowly growing as well).

So I tried to use tokio-console.
I'm able to connect and I see information about my tasks, but I'm very confused because the output seems to be updating every second with new information, even when the test is not running which means my application should be pretty much idle.
There is some minimal housekeeping work triggering in my application but the information about the tasks that I see for 1s on the screen refers to code locations that are executed during the test, and the test is not running.

Is it possible that tokio-console is somehow buffering information and it's not really real-time and is showing me delayed information?

After stopping the test and keeping the application and tokio-console running, tokio-console information seems to become "stable" and not changing every 1s. But it does take several minutes after I stopped the test, which again sounds as if there is a backlog of information tokio-console still needs to process.

Does anybody have a good explanation for this? or any suggestions on how to troubleshoot this further?

1 post - 1 participant

Read full topic