I have some instrumented unit tests with conflicting instrumentation requirements e.g. one requires that a given remote device is connected and another requires that no such device is connected, to make sure I can properly detect the device's presence without false positive or false negative results. These are currently implemented as unit tests for convenience (access to private functions), but could be refactored into integration tests if that would help... but from what I've read in Test Organization - The Rust Programming Language it seems like there is no easy way to run some tests and not others except #[ignore].

Ignore alone doesn't do the trick because in reality there's a matrix of required instrumentation conditions that requires deeper partitioning than just A or (A and B), via --include-ignored. Going back to my detection tests mentioned above as a simple example: I have my #[ignore] on all my instrumented tests since the instrumentation is undefined by default (the test env may or may not have the remote device connected and shouldn't need to worry about it), so cargo test runs fine. However, when I run with --include-ignored, I immediately hit a conflict where one instrumented test needs the device connected to pass and another needs it disconnected. So far I've been working around this using the cargo test test_name_pattern_match -- --include-ignored filter and naming the instrumented tests accordingly to keep them separated, but that seems brittle and precludes running unit tests ignored for a reason unrelated to whatever naming scheme I came up with for a given partition (these simply get filtered out).

Is there a way to cleanly define that I want cargo test to run a defined subset of tests, ideally for arbitrary subsets? Maybe a custom cfg like cargo testA, cargo testB and then accordingly annotated tests? I'm not sure how configurable cargo test is under the hood, and the docs above don't really go into sufficient detail.

5 posts - 2 participants

Read full topic