File not found: How multithreaded doctests involving file creation/destruction can result in intermittent test failures

Rust tests can be run in parallel. Therefore, if multiple tests rely on the same test-related file, and one test deletes the file, it could result in problems for the other tests which are still running.

Let's look at this example which uses doctests. The name of this example library is concurrent_doctests.

src/lib.rs:

/// Removes a file if it exists, ignoring any errors.
pub fn remove_file_if_exists(path: &str) {
    if std::fs::remove_file(path).is_ok() {
        println!("Old file removed.");
    }
}

/// A simple function that writes "hi" to a file.
/// 
/// # Example
/// ```
/// use std::fs::File;
/// use concurrent_doctests::*;
/// 
/// remove_file_if_exists("./output.txt");
/// 
/// let mut file = File::create("./output.txt").unwrap();
/// 
/// write_hi(&mut file).unwrap();
/// 
/// remove_file_if_exists("./output.txt");
/// ```
pub fn write_hi(file: &mut std::fs::File) -> std::io::Result<()> {
    use std::io::Write;
    writeln!(file, "hi")?;
    Ok(())
}

/// A simple function that writes "bye" to a file.
/// 
/// # Example
/// ```
/// use std::fs::File;
/// use concurrent_doctests::*;
/// 
/// remove_file_if_exists("./output.txt");
/// 
/// let mut file = File::create("./output.txt").unwrap();
/// 
/// write_bye(&mut file).unwrap();
/// 
/// remove_file_if_exists("./output.txt");
/// ```
pub fn write_bye(file: &mut std::fs::File) -> std::io::Result<()> {
    use std::io::Write;
    writeln!(file, "bye")?;
    Ok(())
}


/// A simple function that writes "what's up" to a file.
/// 
/// # Example
/// ```
/// use std::fs::File;
/// use concurrent_doctests::*;
/// 
/// remove_file_if_exists("./output.txt");
/// 
/// let mut file = File::create("./output.txt").unwrap();
/// 
/// write_whatsup(&mut file).unwrap();
/// 
/// std::fs::remove_file("./output.txt").unwrap();
/// ```
pub fn write_whatsup(file: &mut std::fs::File) -> std::io::Result<()> {
    use std::io::Write;
    writeln!(file, "what's up")?;
    Ok(())
}

We have four functions:

1. remove_file_if_exists(). Importantly, this function does not panic.
2. write_hi(). This function's doctest has 2 calls to remove_file_if_exists(), and 1 call to File::create().
3. write_bye(). This function and its doctest are essentially the same as that for write_hi().
4. write_whatsup(). On the surface, this function is very similar to the prior 2 (write_hi() and write_bye()), but the difference is that here we introduce the potential for the doctest to panic by calling unwrap() on std::fs::remove_file("./output.txt").

If we run cargo test, we get intermittent test failure for write_whatsup()

Sometimes we get no failures:

$ cargo test
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.01s
     Running unittests src/lib.rs (target/debug/deps/concurrent_doctests-7a17cb407c4bc3e1)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

   Doc-tests concurrent_doctests

running 3 tests
test src/lib.rs - write_whatsup (line 54) ... ok
test src/lib.rs - write_hi (line 11) ... ok
test src/lib.rs - write_bye (line 32) ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

But other times we get a failed test, despite no changes to the underlying code. (It may take a few repeat runs of cargo test to fully recreate the issue.)

$ cargo test
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.01s
     Running unittests src/lib.rs (target/debug/deps/concurrent_doctests-7a17cb407c4bc3e1)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

   Doc-tests concurrent_doctests

running 3 tests
test src/lib.rs - write_bye (line 32) ... ok
test src/lib.rs - write_hi (line 11) ... ok
test src/lib.rs - write_whatsup (line 54) ... FAILED

failures:

---- src/lib.rs - write_whatsup (line 54) stdout ----
Test executable failed (exit status: 101).

stdout:
Old file removed.

stderr:

thread 'main' panicked at /tmp/rustdoctest40yGsV/doctest_bundle_2024.rs:50:38:
called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace



failures:
    src/lib.rs - write_whatsup (line 54)

test result: FAILED. 2 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

error: doctest failed, to rerun pass `--doc`

As you can see, we sometimes get a NotFound error, indicating "No such file or directory" was found. This is because the doctests all rely on a single file "./output.txt", and each construct/destruct it independently.

Solutions

There are several strategies to address this type of issue (if you have solutions not mentioned here, please feel free to comment with them).

One solution is to force the tests to all run in a single thread, one at a time, using a command like one of these:

• cargo test -- --test-threads=1
• RUST_TEST_THREADS=1 cargo test

Another solution is to avoid having tests rely on a single file. Instead of using "./output.txt" everywhere, we could have each doctest use a different file.

Note: This post does have some overlap with this other chat thread, "Race condition in file creation?", so feel free to check that one out too.

1 post - 1 participant

Read full topic