We have a team-internal discussion how one should test functions that use randomness in Rust. So I would like to hear if there are any community-standards / guidelines abut this and if not what are the opinions of community members.

To give a stupid simple example, lets say we have the following function:

fn my_fun(mut vec: Vec<u64>, rng: &mut impl RngExt>) -> u64 {
    ... do some stuff with vec, mutating it in a deterministic way
    vec.shuffle(rng);
    ... process the shuffled vec deterministically, do some stuff with it and compute the end result
}

Now if we want to write some unit tests for this, we have some options:

1. Seed an rng with a fixed seed, then the function should always return the same result and we can assert this result.
2. Use a custom implementation of the Rng trait to "mock" the randomness. This is difficult since you can only implement the Rng methods and not the higher level RngExt methods so you would need to reverse engineer the shuffle implementation to do a low-level mock to achieve exactly the wanted shuffle.
3. Introduce a high level trait for shuffling that can be mocked in the test (i.e. have a trait Shuffler that can shuffle a given vec and instead of impl Rng use impl Shuffler, then the unit test can have a custom implementation of that shuffler trait that is deterministic and predictable by the unit test)
4. Write the test in a way that the asserts only check properties of the result that always are correct regardless what the shuffling did. (depending on the type of the algorithm you might not be able to find many such checks and you might easily miss bugs that still produce valid results according to those properties but doesn't do what you want with the probabilities you want)
5. Do some statistical tests (i.e. run with many different seeds and apply a statistical test to the distribution of the result).
6. Extract the determiinstic parts of the function into private functions (in the example two functions, one for the stuff done before the shuffling and one for the stuff done afterwards), then only test those parts.

That is all the options I could think of for now, but maybe I missed some important ones? Also I guess the answer might depend on the type of randomised algorithm one wants to test (for example if we have some sort of biased sampling, probably a statistical test about the outcome distribution would be helpful, while for other stuff statistical tests might get too complicated) but maybe there are some guidelines to avoid certain things at all times and also when to use what (I guess sometimes also a combination of the above ideas can be useful).

5 posts - 5 participants

Read full topic