Warning

This post was published 118 days ago. The information described in this article may have changed.

Info

This post is auto-generated from RSS feed The Rust Programming Language Forum - Latest topics. Source: What is the expected behavior of `for_each_init` with `par_bridge` in Rayon?

I am experimenting with Rayon’s for_each_init and I noticed some behavior that I want to clarify.

I have a Vec that I iterate using .iter().par_bridge() (which generates an IterBridge) and then call for_each_init. In the init closure I return a Vec<usize>. I observed that:

• While there is data to process, each worker executes init once and reuses that buffer to process items.

• After all data is processed, the init function may be called one or two more times, but the operation closure (op) is not called for these extra invocations.

Example:

use std::{thread, time::Duration};
use rayon::iter::{ParallelBridge, ParallelIterator};

fn main() {
    let v: Vec<usize> = (1..=100).collect();

    rayon::ThreadPoolBuilder::new()
        .num_threads(4)
        .build_global()
        .expect("Error creating thread pool");

    v.iter().par_bridge()
        .for_each_init(
            || {
                println!("Get Vector from worker: {:?}", rayon::current_thread_index());
                Vec::with_capacity(25)
            }, 
            |buffer, n| {
                buffer.push(n);
                if buffer.len() == 25 {
                    println!("Worker: {:?}, Buffer: {:?}", rayon::current_thread_index(), buffer.clone());
                }
                thread::sleep(Duration::from_millis(100));
            }
        );
}

Common output:

Get Vector from worker: Some(2)
Get Vector from worker: Some(0)
Get Vector from worker: Some(1)
Get Vector from worker: Some(3)
Worker: Some(1), Buffer: [...]
Worker: Some(3), Buffer: [...]
Worker: Some(0), Buffer: [...]
Worker: Some(2), Buffer: [...]
Get Vector from worker: Some(1)

However, when I use .par_iter().for_each_init() I notice that init is executed multiple times per worker during processing.

So, my question is: Does for_each_init with par_bridge guarantee that each worker executes the init closure at most once while processing all data, or is it possible that it may be called multiple times per worker, similar to .par_iter().for_each_init()?

1 post - 1 participant

Read full topic

🏷️ Rust_feed