The Rust for Linux project
Why Rust?
This year marks the 30th anniversary of the Linux kernel, he began, but it is also the 30th anniversary of the first ISO standard for the C programming language. C has a lot of history and a lot of baggage, but it is still relevant and interesting. Ojeda said that he is working with the C committee to improve the language, but that process will take a long time — if it ever reaches fruition.
C is good for kernel development for a number of reasons. It is fast,
facilitates low-level programming, is simple, and fits the problem domain well.
But there is one little problem, known generally as "undefined behavior".
The advantage of Rust is that it eliminates undefined behavior, at least in
"safe" code. Indeed, the lack of undefined behavior is what Rust developers
mean when they say "safe"; it has no relation to similar terms like
"safety-critical". An abort() call in C is safe by this
definition, even if the end result kills somebody. Ojeda has proposed a
"[[safe]]" attribute for C to mark functions that are written to
avoid all undefined behavior (and have the compiler enforce that),
but that proposal has not made much progress.
So what is not "safe" by Rust standards? The list of not-safe behaviors include using pointers after they are freed, dereferencing NULL pointers, freeing memory twice, using the contents of uninitialized memory, out-of-bounds memory accesses, data races, and more. None of these things will happen, he asserted, in safe sections of Rust code; that is why he wants to see Rust code in the kernel. About 70% of published C vulnerabilities result from undefined behavior, he said, and the rate for vulnerabilities in the Android media and Bluetooth components is closer to 90%. Rust can help make those problems go away. Rust offers a number of other advantages as well, including stricter types, modules, pattern-matching primitives, lifetimes, an extensive set of development tools, and much more.
What is the downside of using Rust in the kernel? It is not possible to model everything, he said, so it is still necessary to use some unsafe code. The Rust language is much more complex than C, and its extra run-time checks can hurt performance. And, of course, it is another language for kernel developers and maintainers to learn, which is also a significant cost. It's a one-time cost that, he said, pays off, but it's still a cost. The Rust for Linux developers are happy to help developers get past this initial learning obstacle.
So how does Rust compare to C as a systems-programming language? Like C, it is fast. It's somewhat low-level like C but, depending on the code, it might be less easy for developers to predict what the resulting assembly code will look like. Rust is definitely not simple like C is. He believes that it fits the domain as well as C does, but others may differ.
Rust support in the kernel
There are five supported architectures for Rust so far: armv6, arm64, powerpc, riscv, and x86-64. Work to this point has not been aimed at supporting every possible architecture; that is a lot of low-level work that doesn't necessarily demonstrate anything new. Instead, the intent is to show that supporting a variety of architectures is possible. There are currently three projects working on compiling Rust code for the kernel. Two of them use the "official" rustc front-end, then using LLVM or GCC for code generation. The rustc/LLVM combination is the leading compiler at this point; Ojeda did not say much about the rustc/GCC pairing. There is also a project working on a native Rust compiler for GCC (gccrs); that is expected to be ready in a year or two.
The Rust for Linux work is split between the Rust and kernel trees. The Rust side has two crates called core (low-level functionality) and alloc (data structures and such). The alloc crate is part of the submission for the kernel side as well; that might eventually not be necessary. These crates are considered to be more a part of the standard Rust library than the kernel.
On the kernel side there are the kernel and builtins crates. The kernel crate abstracts the interface to the rest of the kernel. The bindgen tool is used to generate the bindings that allow calling kernel functions from Rust; there is currently no provision for calling back into Rust from C. One problem that still needs a complete solution, he said, is keeping the C and Rust sides synchronized. The intent is for Rust to be a first-class citizen in the kernel; if a developer makes a change that breaks Rust code, they should fix it, just as they would with the kernel's C code. Just how that will work is not yet fully clear, though.
Where things get interesting is the driver author's point of view. It would be possible to create Rust bindings for all of the kernel functions that one normally calls from a driver and have it work, but that is not what the Rust-on-Linux developers are trying to do. Instead, they are populating the kernel crate with a set of subsystem abstractions and interfaces that will, it is hoped, make it possible to write drivers with no unsafe code at all, he said.
Discussion
There were some concerns raised at this point about the need for maintainers to learn Rust. Ojeda acknowledged that this will be necessary; in the end, maintainers need to be able to take responsibility for their subsystems, even if the Rust developers help them initially. Laurent Pinchart said that help could be needed for a long time; he does not have the time to stop everything and learn Rust anytime soon, so he will be hard put to take responsibility for any Rust code in his subsystem.
Ojeda answered that the Rust developers are aware of this; they have been
trying to pick subsystems to start that have maintainers with the time and
interest to take up this challenge. Hopefully that will result in the
creation of enough examples and experience to help bring in other
subsystems later. He reiterated, though, that maintainers need to be on
board; if Rust isn't a first-class citizen in the kernel, this experiment is
not going to work. Half in jest (perhaps), he added that maintainers who
jump on the Rust train sooner are likely to get more help from the Rust
developers than those who come later.
Ojeda said that, while subsystem maintainers may have some difficult work to do, life will be easier for driver writers, who only need to work with the (safe) interfaces provided to them. Pinchart objected that driver writers often find that they need to make core-subsystem changes to support their devices properly, so life isn't quite that simple. If the bar for driver authors is raised to include being fluent in Rust, that will be a real impediment for some, he said.
Paolo Bonzini worried that Rust code could inhibit changes to internal kernel APIs. A developer may find that Rust code is the only remaining user of a given API, but may lack the desire (or the ability) to fix that code, so the old API will remain. He asked: how many wrappers for core-kernel APIs do the Rust developers plan to add? If, for example, only some file_operations members are used by Rust code, will there still be bindings for all of them? Adding more bindings than necessary may make it harder to change those operations in the future. Ojeda replied that, for something as core as file_operations, they might just add them all, since they will be used eventually. In other cases, only the interfaces actually used by Rust code will be implemented.
The conversation wandered into the toolchain; Ojeda said that the Rust for Linux developers are currently only supporting one version of the Rust compiler with any given kernel version. Others may work, but there is no guarantee. Eventually it will be possible to build the kernel using only stable Rust features, at which point it will be possible to support more compiler versions. He added that concerns about the stability of the Rust language in general have been raised, but he does not think that is a problem. The language is stable now, he said, and becoming more stable quickly. Kernel code written now, even code using unstable features, will "mostly" work going forward.
Philip Herron, who is working on gccrs, asked: what areas in particular would the Rust for Linux developers like to see written in Rust? Ojeda suggested filesystems and other subsystems that contain a lot of state would be good candidates. Herron then asked about whether there were any plans to pick a specific subset of the Rust language to use in the kernel. He asked specifically about const generics which, he said, would be a good feature to use in the kernel context. But will there come a time when the use of language features must be limited to reduce the burden on maintainers? Ojeda said that some of the more esoteric features might be useful in the core code, but he would rather see drivers using a smaller subset of the language and avoid things like "crazy metaprogramming" and such.
At this point, the session ran out of time and the participants headed off
for a much-needed coffee break; further discussion was deferred to the
following two days. More than 30 developers attended the session,
indicating that there is a fair amount of interest in the idea of writing
kernel code in Rust, even if that capability remains out of the mainline
kernel for now. Rust in the kernel is not an easy sell, but if it fails, it
will not be due to a lack of trying.
Index entries for this article | |
---|---|
Kernel | Development tools/Rust |
Conference | Kangrejos/2021 |
Posted Sep 16, 2021 17:54 UTC (Thu)
by atnot (subscriber, #124910)
[Link] (4 responses)
Posted Sep 16, 2021 22:17 UTC (Thu)
by tux3 (subscriber, #101245)
[Link] (3 responses)
I imagine there's a lot of scaffolding, core code, and example drivers to write. Perhaps that's why it takes more than a couple months for them to move to the next step.
If you feel they should abandon their efforts, or not organize conferences to talk about the project's status, you should certainly want to share a good reason why. I think everyone can agree that we'd all like to hear novel discussion points.
Posted Sep 17, 2021 2:27 UTC (Fri)
by atnot (subscriber, #124910)
[Link] (2 responses)
Posted Sep 17, 2021 7:40 UTC (Fri)
by beagnach (guest, #32987)
[Link] (1 responses)
Posted Sep 23, 2021 4:11 UTC (Thu)
by marcH (subscriber, #57642)
[Link]
Posted Sep 16, 2021 18:25 UTC (Thu)
by pj (subscriber, #4506)
[Link] (5 responses)
Posted Sep 16, 2021 18:29 UTC (Thu)
by corbet (editor, #1)
[Link] (3 responses)
Posted Sep 16, 2021 22:38 UTC (Thu)
by Wol (subscriber, #4433)
[Link] (2 responses)
If they start off writing code that will only be run on supported platforms, it doesn't need to run on the others.
If there's a "rust to C" translator (I believe there is), then they can use that for unsupported platforms.
And as rust gains a foothold, it will provide pressure for the list of supported platforms to grow.
These things don't burst on the scene fully-fledged - linux didn't, why should rust?
What's that saying? If you don't evolve you die? Maybe rust is the next evolution for linux? Maybe it isn't, but don't put artificial obstacles in the way - you should know linux well enough to know it will evolve around and past you if you try to set up a road-block.
Cheers,
Posted Sep 16, 2021 22:39 UTC (Thu)
by Wol (subscriber, #4433)
[Link]
Cheers,
Posted Sep 21, 2021 6:00 UTC (Tue)
by flussence (guest, #85566)
[Link]
Most old/exotic platforms aren't plug-and-play anyway; even if every new peripheral and PCIe card going forward had a pure-Rust driver it's unlikely to affect anyone on m68k, mips, alpha etc.
(The kernel's already got plenty of platform-limited compiler-dependent stuff anyway: LLVM LTO and GCC hardening plugins come to mind - those two are even mutually exclusive!)
Posted Sep 18, 2021 11:22 UTC (Sat)
by federico3 (guest, #101963)
[Link]
Posted Sep 16, 2021 23:47 UTC (Thu)
by roc (subscriber, #30627)
[Link] (13 responses)
Posted Sep 17, 2021 9:42 UTC (Fri)
by pbonzini (subscriber, #60935)
[Link] (8 responses)
As a maintainer (not as a kernel maintainer), my main worry in adopting Rust is: will I be able to review code effectively? will I correctly identify neverending sources of future technical debt and stop them from entering the project?
I like the language, but despite the extra safety guarantees offered by the compiler, I find Rust code to be very hard to review, even more so than C++.
Posted Sep 17, 2021 16:16 UTC (Fri)
by moltonel (guest, #45207)
[Link] (7 responses)
Posted Sep 17, 2021 16:42 UTC (Fri)
by pbonzini (subscriber, #60935)
[Link] (6 responses)
Almost every time I wrote complex code in Rust, I ended up throwing away the first version because I went down the wrong way to define types, establish ownership policies, pick what was hardcoded vs. what was abstracted. And it took a while to realize that, possibly only after using the library in anger. That is what I found even worse in Rust than in C++.
Posted Sep 17, 2021 23:39 UTC (Fri)
by st33med (guest, #144794)
[Link]
In C, I would argue many things are presented as 'overly simplified' with hidden behaviors that many developers may not consider until it bites you with undefined behavior. The typing system is not very strict which gives many footguns, there's practically no such thing as memory safety, etc. I think you know all this, and you're right that Rust adds features to be safe, and programming under Rust with this subset takes some time to think about how you want a program to function.
What I'm getting at is that it takes time and experience with Rust to be able to formulate an effective and friendly program structure, just like many other programming languages. You wouldn't be expected to pick up Haskell and know exactly the 'right' way to make a web server application, and Rust is no different. There's also never one obvious way to construct a program; sometimes you do need to scrap your current code and start over, regardless of the language used, and that happens less as you learn about the tools you are using.
The additional features of Rust do add more to consider in your design and may add to technical debt later on, but with experience, you gain insight into how your program synergizes with the Rust model and it becomes less of a burden. Likewise, the clearly marked unsafe portions of code, enforced typing model, and lack of undefined behavior remove a lot of time reviewing code and running static checkers against your program, of which C++ does not really remove despite being more 'feature-rich' than C and containing things like templates that Rust doesn't have.
Posted Sep 18, 2021 0:06 UTC (Sat)
by roc (subscriber, #30627)
[Link] (2 responses)
Seems to me your concern applies mainly to green-field development in Rust. Actual *maintenance* presents these questions less often, and when it does the solutions will be constrained by the patterns already set up.
But it would be interesting to collect more specific examples and see what can be done to provide guidance. I personally don't feel this burden myself. Maybe it's because I've been writing Rust code for five years, but even when I look at the code I wrote in the first year I think the design generally holds up.
Posted Sep 18, 2021 3:52 UTC (Sat)
by pbonzini (subscriber, #60935)
[Link] (1 responses)
Posted Sep 18, 2021 9:00 UTC (Sat)
by laarmen (subscriber, #63948)
[Link]
However, I often find pure Rust code patches easier to review than pure C patches, assuming knowledge of the underlying code base, as Rust allows for stronger local reasoning. If the patch introduces a call looking like foo(&my_struct), Rust tells me that the contents of my_struct won't change (barring interior mutability) whereas, seeing the exact same call in C, I'll need to know the particulars of foo() in order to judge the soundness of further use of my_struct.
Posted Sep 20, 2021 15:17 UTC (Mon)
by moltonel (guest, #45207)
[Link]
I understand the fear, but it doesn't match my experience.
First the most obvious: paying technical debt (aka refactoring) is easier in Rust, because an API change will be noticed by the compiler in all callers, whereas in C many would be wrongly ignored. I rarely get the "needs a refactor but I'm too afraid to break things so I'll leave it" feeling with Rust.
Second, Rust helps making APIs that are harder to misuse, actively fighting technical debt. That's what you encountered when you realized you needed to throw away your first version(s). You probably should throw away your first C version too, but the compiler and base APIs don't tell you that. In a way, rustc is your first reviewer, making the human reviewer's job simpler. It might cause some anger when you're still learning that API or Rust itself, but it seems worth it.
Posted Sep 23, 2021 4:54 UTC (Thu)
by marcH (subscriber, #57642)
[Link]
Define "worse", on which axis?
Debugging undefined behaviors and race conditions or refactoring existing code? I have my preference...
Posted Sep 29, 2021 10:20 UTC (Wed)
by lacos (guest, #70616)
[Link] (3 responses)
In any busy project, maintainers tend to operate at 100+% of their capacity. Whenever they need to learn something new, that effort is taken away from their "usual" activities (which are possibly the activities that they get paid for). Obviously they keep learning new things incrementally, but new material that's intrusive and has a steep learning curve (= late RoI) must be a different sell.
> I guess the problem is that those changes have been incremental but "learn Rust" is not.
IMO: exactly so.
Posted Sep 29, 2021 11:22 UTC (Wed)
by Wol (subscriber, #4433)
[Link]
I'm now working with Oracle rather than Pick, so switching from ENGLISH to SQL is horrible. SQL is just so much more convoluted, slow, horrendous ... (and I've just managed to speed up a query from 10 mins to 2, just by thinking hard about how it was written. The Pick version would probably have taken one minute as the first draft!).
Then there's learning VBA. Okay, I know assorted programming languages so this isn't so bad ...
But it gets wearing when pretty much everything you want to do results in a google query for "what's the syntax for doing XYZ?". And working remotely, I can't even easily ask my co-workers ...
Cheers,
Posted Oct 1, 2021 20:27 UTC (Fri)
by bfields (subscriber, #19510)
[Link]
Compared to the stuff I'm usually fighting to swap in (how does the dcache work again?), I feel like a new language isn't too bad. They're well-documented. They usually share a lot with other languages. The basics you need to get started usually just aren't that complicated.
Posted Oct 2, 2021 21:40 UTC (Sat)
by neilbrown (subscriber, #359)
[Link]
Then they need to delegate. Or abdicate.
Posted Sep 17, 2021 6:45 UTC (Fri)
by kaviraj (subscriber, #136156)
[Link] (3 responses)
Posted Sep 17, 2021 13:26 UTC (Fri)
by corbet (editor, #1)
[Link] (2 responses)
Posted Sep 17, 2021 16:06 UTC (Fri)
by ojeda (subscriber, #143370)
[Link]
Posted Sep 17, 2021 19:32 UTC (Fri)
by ojeda (subscriber, #143370)
[Link]
Posted Sep 17, 2021 9:28 UTC (Fri)
by gray_-_wolf (subscriber, #131074)
[Link] (8 responses)
If the other subsystem is (partially) in rust, which you do not know, how would that be handled? If some parts of your patch touch some rust code, but goes via a tree of a maintainers that does not know rust, how should that be handled?
Just to make it clear, I really am just trying to understand how this will (would?) work. These are really obvious questions, so I've likely just overlooked it already being discussed somewhere.
Posted Sep 17, 2021 9:47 UTC (Fri)
by pbonzini (subscriber, #60935)
[Link] (4 responses)
But apart from that, for something such as drivers (which are the initial focus of the project), the cases of patches affecting multiple subsystems are quite rare.
Posted Sep 17, 2021 15:21 UTC (Fri)
by broonie (subscriber, #7078)
[Link] (3 responses)
Posted Sep 17, 2021 15:56 UTC (Fri)
by pbonzini (subscriber, #60935)
[Link] (2 responses)
Posted Sep 18, 2021 8:24 UTC (Sat)
by matthias (subscriber, #94967)
[Link] (1 responses)
Posted Sep 18, 2021 18:32 UTC (Sat)
by broonie (subscriber, #7078)
[Link]
Posted Sep 17, 2021 16:41 UTC (Fri)
by ojeda (subscriber, #143370)
[Link] (1 responses)
No, they don't. At the moment, the Rust side is a client of the C APIs, and thus the modifications to the C side are minimal.
The problem is that, even in this case, if you are doing subsystem or tree-wide changes that change C APIs, and they happen to be used by Rust code, then you would also need to update those.
This is what some kernel maintainers/developers that regularly do those kind of changes are worried about -- they would need to learn Rust fairly soon, which may put a big burden on them. Thus we may need to have a way to smooth things for them (e.g. relaxing the rules, helping them write or review the Rust side of the code, etc.).
Posted Sep 17, 2021 16:58 UTC (Fri)
by ojeda (subscriber, #143370)
[Link]
Ignore this sentence -- I thought you were referring to the Rust changes. For kernel changes in general, yes, some of them change different places in the kernel (the second part of the comment addresses that).
Posted Sep 17, 2021 19:00 UTC (Fri)
by ojeda (subscriber, #143370)
[Link]
No, I don't think you overlooked :) We have talked about these points a few times, but they are still quite open. At LPC next week we will try to have this discussion with more kernel maintainers.
Posted Sep 23, 2021 4:15 UTC (Thu)
by marcH (subscriber, #57642)
[Link] (3 responses)
I really doubt anyone can predict the assembly code produced from any non-trivial piece of C unless they turn off optimizations.
https://queue.acm.org/detail.cfm?id=3212479 "C is not a low-level language", etc.
Posted Sep 24, 2021 14:37 UTC (Fri)
by error27 (subscriber, #8346)
[Link] (2 responses)
Some of the kernel is written in assembly. But there are definitely parts which are written in C but the author has the assembly in mind and reviews the assembly to verify.
Posted Sep 26, 2021 8:08 UTC (Sun)
by marcH (subscriber, #57642)
[Link] (1 responses)
Posted Sep 27, 2021 13:06 UTC (Mon)
by mathstuf (subscriber, #69389)
[Link]
The Rust for Linux project
The Rust for Linux project
The Rust for Linux project
The Rust for Linux project
The Rust for Linux project
The Rust for Linux project
Rust does not yet work on all of the architectures supported by the Linux kernel; this was discussed a bit more in the later days (stay tuned). That's recognized as a problem and is one of the reasons why people are looking forward to the native GCC Rust compiler.
Architecture support
Architecture support
Wol
Architecture support
Wol
Architecture support
The Rust for Linux project
The Rust for Linux project
The Rust for Linux project
The Rust for Linux project
The Rust for Linux project
The Rust for Linux project
The Rust for Linux project
The Rust for Linux project
The Rust for Linux project
The Rust for Linux project
The Rust for Linux project
The Rust for Linux project
The Rust for Linux project
Wol
The Rust for Linux project
In any busy project, maintainers tend to operate at 100+% of their capacity. Whenever they need to learn something new, that effort is taken away from their "usual" activities (which are possibly the activities that they get paid for).
The Rust for Linux project
Such behaviour is not good for the mental health of the maintainer or the long term health of the project. Just don't do it.
If you stop, that creates a vacuum. Vacuums tend to get filled.
The Rust for Linux project
My understanding is that recordings for a subset of the talks will be posted.
Recordings
Recordings
Recordings
How will cooperation of rust-knowing and not-rust-knowing maintainers look like?
How will cooperation of rust-knowing and not-rust-knowing maintainers look like?
How will cooperation of rust-knowing and not-rust-knowing maintainers look like?
How will cooperation of rust-knowing and not-rust-knowing maintainers look like?
How will cooperation of rust-knowing and not-rust-knowing maintainers look like?
How will cooperation of rust-knowing and not-rust-knowing maintainers look like?
How will cooperation of rust-knowing and not-rust-knowing maintainers look like?
How will cooperation of rust-knowing and not-rust-knowing maintainers look like?
How will cooperation of rust-knowing and not-rust-knowing maintainers look like?
The Rust for Linux project
The Rust for Linux project
The Rust for Linux project
The Rust for Linux project