I'm reading the neat rust-cli guide for making cli apps with clap.

I'm almost certainly a misunderstanding from my part, but here is what I find confusing:

There, there is this paragraph:

Exercise for the reader: This is not the best implementation as it will read the whole file into memory, no matter how large the file may be. Find a way to optimize it! (One idea might be to use a BufReader instead of read_to_string().)

The BufReader docs say:

BufReader<R> can improve the speed of programs that make small and repeated read calls to the same file or network socket. It does not help when reading very large amounts at once, or reading just one or a few times. It also provides no advantage when reading from a source that is already in memory, like a Vec<u8>.

So I reasoned this way:

1. If a file is loaded with read_to_string or read, then we have the file in memory as String or Vec<u8> respectively.
2. Then using BufReader won't help, as they say in the paragraph.

Probably 1. is not a single syscall, but none of the docs for read_to_string or read state anything related to performance.

While writing I thought, maybe there is some trait they implement and has this info (like Read). It says:

Please note that each call to read() may involve a system call, and therefore, using something that implements BufRead, such as BufReader, will be more efficient.

But aren't we calling .read/read_to_string just once? Possibly not, but I am unsure. I believe a part of my confusion arises from:

1. There is the trait Read and its method read
2. There are the implementors, which can use it to read any size.
3. This means the implementors may choose to read a few bytes and then we need many syscalls to read an entire file with fs::read_to_string or fs::read.

At this point I asked some Chatbots and the answer seems to be that fs::read does perform many small syscalls in a loop.

So I assume the guide is correct, and by using BufRead we guarantee using an implementation of Read that has a larger chunk size and hence performs fewer syscalls.

So is the guide's claim correct? Any conceptual gap you'd fill in, in my description?

Maybe I should look at why fs::read may do many syscalls in the source code myself but I fear doing it since it may be quite hard.

3 posts - 2 participants

Read full topic