Note: this question is similar to this one from 2017, but that thread doesn't have a clear answer and I'm hoping for something more informative (hopefully there's more knowledge about this, 8 years later).

At work, I started helping with a Rust project that relies on the C library libpng to deal with images. I'm trying to convince myself we are using the library in a correct way, without triggering UB. I'd love to have a second opinion though, because unsafe Rust is notoriously difficult to get right.

The main challenge is that libpng handles errors using setjmp and longjmp (in a way that resembles throwing and catching exceptions). The user is responsible of using setjmp at the right places, and libpng is responsible for calling longjmp when an error is encountered. In Rust, however, setjmp and longjmp are heavily discouraged (I'm not even certain you can use them at at all without some hackery), so we'd rather not use them.

Fortunately, libpng allows us to configure custom error callbacks. That means that, with the right callbacks, we can skip setjmp / longjmp entirely and rely on Rust's unwinding instead to propagate errors. We currently provide callbacks that trigger an unwind on error, and we make sure to catch the unwind somewhere up the stack. Here's an example:

// Fatal error handler callback
unsafe extern "C-unwind" fn err_handler(_png_ptr: *mut ffi::png_struct, _msg: *const c_char) {
    resume_unwind(Box::new(()));
}

// Read callback (we are reading from memory)
unsafe extern "C-unwind" fn read_handler(png: *mut ffi::png_struct, data: *mut u8, len: usize) {
    let data = slice::from_raw_parts_mut(data.cast(), len);
    let reader = &mut *ffi::png_get_io_ptr(png).cast::<Cursor<&[u8]>>();
    if reader.read_exact(data).is_err() {
        // Unexpected EOF, trigger unwind
        resume_unwind(Box::new(()));
    }
}

// Rust function to decode an image
pub(crate) fn decode_png(data: &[u8]) -> Result<DecodedImage> {
    let result = catch_unwind(|| {
        // Code that calls libpng functions that may unwind
    });
    result.unwrap_or(Err(Error::DecodeFailed))
}

With this in place, everything seems to work properly (fingers crossed). However, people in the Rust community often mention it's unsafe to trigger an unwind from Rust, let it go through a C library and catch it when it arrives back to Rust code (meaning that it is undefined behavior). Because of that, I'm in doubt: how can I be sure what I'm doing satisfies Rust's safety requirements? If it's impossible with the current approach, do you know of any alternative that would guarantee safety?

2 posts - 2 participants

Read full topic