I have a server application that can be configured to listen for incoming client connections. It supports both plain tcp and tls (using tokio-rustls).

The issue I'm running into is that while the plain tcp interface seems to work without issues (but I can't be certain that is the case) the tls works fine for a while (on average it seems to work for a day or so (probably doing between 10-40 connections in one day)), but then the listener simply stops accepting incoming connections (client gets an immediate error 61 (macos, but I think this is a pretty universal unixy error code) trying to connect). Here's why it confuses me: The listener is just a loop that accepts an incoming connection, and immediately spawns a new task (using the multithreaded tokio runtime) to handle the connection. I've seen this sort of behavior when I accidentally left an unwrap() in the listener task, but I don't see any panics and I have searched through the code for any left-over unwrap()/expect().

In fact, I don't see any errors at all(!), and I'm not ignoring any Results. That's what's most confusing -- it behaves like the listener task would have panic'd, but there's no trace that such a thing would have happened.

Due to the lack of any kind of warning/error, I not sure how I should approach this. I would like to check if the listener task is still alive when these problems occur. I could add some kind of heartbeat and a watchdog, but are there any tools that can help debug these kinds of issues without needing a bunch of temporary code changes? Is this something tokio-console can be used to investigate?

1 post - 1 participant

Read full topic