I have a gRPC server built with Tonic that is exhibiting the terrible behavior of getting into a state where it is running but not handling any requests.

By "handling" I mean that the Tonic service methods are not even called when the server is hit. The first line of each such method is a trace (info!) that is never printed. Clients requests time out, and the server shows no sign of being hit. (Yet it is bound to the IP, and tcpdump on the VM shows the incoming request.)

There is no memory leak nor high CPU usage: When this occurs, RAM is <5Mb as always, as shown by htop, and CPU 0%.

There are no errors preceding the state - nothing logged (every error condition has a log, using tracing), and nothing in the stdout/stderr redirect.

I am worried this is some bug in Tonic.

The server is constructed and run like this:

fn main() -> Result<(), Box<dyn Error>> {
    let runtime = tokio::runtime::Builder::new_multi_thread()
        .enable_all()
        .build()
        .expect("Failed to construct Tokio Runtime");

    ...

    match runtime.block_on(
        Server::builder()
            .add_service(service1::new())
            .add_service(service2::new())
            .serve(address),
    ) {
        Ok(_) => Ok(()),
        Err(e) => {
            error!(".serve() error: {}", e);
            Err(Box::new(e))
        }
    }

The only other thing possibly noteworthy about the app is that it bridges non-async code with this utility:

    pub fn run<F, R>(f: F) -> R
    where
        F: Future<Output = R>,
    {
        task::block_in_place(move || runtime::Handle::current().block_on(f))
    }

This is always called from the context of the Runtime created above (this is the only Runtime created in the app).

Any ideas on things to check, etc.? I realize this is not a complete description of the app - I do no have a minimal reproducible sample - but I think it is all the pertinent information.

Is there anything in general that can cause behavior like this from tonic/tokio?

1 post - 1 participant

Read full topic