Warning

This post was published 33 days ago. The information described in this article may have changed.

Info

This post is auto-generated from RSS feed The Rust Programming Language Forum - Latest topics. Source: Designing a Rust Kernel with Loadable Modules: Is this "Strict Versioning + LTO Hack" architecture theoretically sound?

Hi everyone,

First of all, thank you so much for taking the time to read this.

I am currently in the design phase of a toy monolithic kernel in Rust. I am trying to implement support for Loadable Kernel Modules (LKMs).

I haven't started the implementation yet because I want to validate the theoretical soundness of my architecture before I go down a rabbit hole. I must admit that my understanding of rustc internals and LLVM behavior is limited. I have built this design based on my current understanding, but I am worried that it might rely on undefined behaviors that happen to work by coincidence, rather than solid principles. I would incredibly value your expertise on this.

The Context (Constraints & Alternatives):

1. No Stable ABI needed: I am not trying to support modules compiled with different rustc versions. The Kernel and Modules are always compiled in lockstep (exact same compiler, exact same flags).
2. Ergonomics first: I want to avoid #[repr(C)] or Opaque Pointers everywhere. I want to use native Rust structs and Traits as much as possible.
3. No Generics: I assume the strategies below cover my needs, so I do not plan to support exporting generic functions across the boundary.
4. Why not existing crates? I am aware of crates like thin_trait_object, abi_stable, and safer_ffi. However, I find them too "heavy" for my kernel context. I aim for a minimal solution that leverages the compiler's behavior rather than introducing complex abstraction layers or dependencies.

The Proposed Architecture:

I plan to split the project into a library (ABI), the kernel (Runtime), and the modules.

/workspace
│
├── kernel_abi/ (Lib, crate-type="rlib")
│   └── Defines structs (repr-Rust), traits, and interface functions.
│       Pure logic/interface, NO static state.
│
├── kernel/ (Bin)
│   └── Statically links `kernel_abi`.
│       Holds global state (allocator, etc.).
│       Contains my custom ELF Loader.
│
└── driver_net/ (Lib, emits ".o")
    └── Depends on `kernel_abi`.
        Acts as a leaf node (device driver).

The Intended Workflow:

1. kernel_abi is compiled once to libkernel_abi.rlib.
2. kernel links libkernel_abi.rlib (with LTO enabled).
3. module compiles using metadata from libkernel_abi.rlib but emits a relocatable object (.o). (LTO disabled).
4. Runtime: My kernel loads the module's .o file, parses it, and manually resolves undefined symbols (U) against the kernel's internal symbol table.

My Hypotheses & Questions:

1. Structs & Layout (The "Determinism" Gamble)
I plan to use default #[repr(Rust)] for most structs in kernel_abi.

• My Hypothesis: Since both the Kernel and the Module compile against the exact same rlib metadata and use the exact same compiler version, the memory layout (field offsets) should be deterministic and identical in both artifacts.
• The Goal: If this holds true, I hope to pass &MyStruct across the boundary safely without forcing repr(C).

2. Function Boundary (No Inlining)
This applies to both standard impl methods and statically dispatched trait methods (e.g. obj.trait_method()) called by the module but implemented in the kernel.
I intend to use a macro to enforce #[inline(never)] and #[export_name].

• Intended Mechanism: The module generates a standard call SymbolName.
• Relocation: My kernel loader handles patching these calls at runtime to point to the implementation inside the kernel.

3. Dynamic Traits & VTables (The LTO Nightmare)
I need to pass dyn Trait objects bidirectionally across the boundary.While Module-to-Kernel calls can often be handled via static dispatch (exported symbols), the critical path is Kernel-to-Module calls (e.g., the Kernel calling driver.read() on a trait object provided by the Module).

The possibility of LTO in the Kernel silently reordering VTable entries gives me a massive headache. I am worried that LTO might perform Virtual Function Elimination (VFE) or reorder pointers based on usage, causing a fatal layout mismatch with the Module (which generates a full VTable).

• My Guess: If I compile the Kernel with -C link-arg=-export-dynamic, I suspect this might force LTO to preserve all public symbols (including VTables) intact, because the linker assumes they might be used externally.
• The Proposed Hack: Since I don't actually want the bloat of dynamic symbol tables in my raw kernel binary, I plan to discard the .dynsym section in the Kernel's Linker Script (/DISCARD/).

I am seeking validation on these points:

1. Is my assumption about #[repr(Rust)] layout determinism theoretically safe? (Given: exact same compiler + shared rlib metadata).

2. Does -export-dynamic actually stop LLVM/LTO from altering VTable layouts (e.g. trimming unused methods)? Or am I misunderstanding how LTO interacts with exported symbols?

3. The Dealbreaker: Does LLVM/LTO ever reorder VTable entries for #[repr(Rust)] traits? * Context: If I can ensure no entries are trimmed (via Q2), can I assume the order remains deterministic (e.g., declaration order)? If LTO reorders VTables, my kernel will explode and I will abandon this approach immediately.

I sincerely appreciate any guidance you can provide. I want to know if this architecture is sound in principle, or if I should stop trying to be clever and stick to repr(C). Thank you!

(P.S. Please forgive the AI-like formatting of this post. Trying to organize all these architectural details into readable text manually is a disaster, so I used some help to clean it up! :-))

1 post - 1 participant

Read full topic

🏷️ Rust_feed