I found many threads discussing the fact that file backed mmap is potentially unsafe, but I couldn't find many resources about shared memory with MAP_ANON. Here's my setup:

Setup details:

• I use io_uring and a custom event loop (not Rust async feature)
• Buffers are allocated with mmap in conjuction with MAP_ANON| MAP_SHARED| MAP_POPULATE| MAP_HUGE_1GB
• Buffers are organized as a matrix: I have several rows identified by buffer_group_id, each with several buffers identified by buffer_id. I do not reuse a buffer group until all pending operations on the group have completed.
• Each buffer group has only one process writing and at least one reader process
• Buffers in the same buffer group have the same size (512 bytes for network and 4096 bytes for storage)
• I take care to use the right memory alignment for the buffers
• I perform direct IO with the NVMe API, along with zero copy operations, so no filesystem or kernel buffers are involved
• Each thread is pinned to a CPU of which it has exclusive use.
• All processes exist on the same chiplet (for strong UMA)
• In the real architecture I have multiple network and storage processes, each with ownership of one shard of the buffer, and one disk in case of storage processes
• All of this exists only on linux, only on recent kernels (6.8+)

IPC schema:

• Network process (NP) mmap a large buffer ( 20 GiB ?) and allocates the first 4 GiB for network buffers
• Storage process (SP) gets the pointer to the mmap region and allocates the trailing 16 GiB as disk buffers
• NP receive a read request, and notify storage that a buffer at a certain location is ready for consumption via prep_msg_ring (man page)
• SP parse the network buffer, and issue a relevant read to the disk
• When the read has completed, SP messages NP via prep_msg_ring that a buffer at a certain location is ready for send
• NP send the disk buffer over the network and, once completed, signals SP that the buffer is ready for reuse

Questions:

• Is this IPC schema safe?
• Should I be worried about UB?
• Is prep_msg_ring enough of a synchronization primitive?
• How would you improve this design?

5 posts - 3 participants

Read full topic