The following example reads a file in a streaming manner, rather than loading it entirely, and calculates the sha256 checksum.

use std::{error::Error, fs::File, io::Read};

use sha2::{Digest, Sha256};

fn main() -> Result<(), Box<dyn Error>> {
    let mut reader = File::open("Cargo.toml")?;
    let mut hasher = Sha256::new();
    let mut buffer = [0u8; 8192];

    loop {
        let n = reader.read(&mut buffer)?;
        if n == 0 {
            break;
        }
        hasher.update(&buffer[..n]);
    }

    let checksum = hasher.finalize();
    println!("{:x}", checksum);

    Ok(())
}

Should I use std::io::BufReader like this?

let file = File::open(path)?;
let mut reader = BufReader::new(file);

The buffer in the example above is 8192 bytes.
I'm working on Ubuntu, so BufReader::new(file) also uses the same size for its inner buffer, since it uses DEFAULT_BUF_SIZE, which is defined as follows:

pub const DEFAULT_BUF_SIZE: usize = if cfg!(target_os = "espidf") { 512 } else { 8 * 1024 };

I'm concerned that using two buffers of the same size might introduce unnecessary overhead.

I think modern operating systems cache loaded files in RAM, and I don't know the correct way to write benchmarks that measure I/O performance without being influenced by such caching mechanisms.

2 posts - 2 participants

Read full topic