rdevd a devd re-implemented on Rust

eesekaj

eesekaj

Hello,

I am a developer of freebsd-kpi-rs and freebsd-kmi-rs (R4FBSDk - Rust for FreeBSD kernel). Recently I have released the devd(8) re-implemented in Rust (ported from closed source code which I developed for the client) and I was planning to submit port for review. But, for reasons beyond my control, I decided to terminate both projects and focus on the new projects i.e RISC-V/ARM bootloader (Rust) and microkernel (Rust) with userland process schedulers, for the embedded devices i.e routers, SCADA etc. Both are focused on European market only, so in US you won't hear about it a lot.

I am not going to maintain the code anymore, so if you really need it, you can download it from the floowing repos (access is not available from some countries).
1) rdevd a devd device state change daemon which is 100% compatiable with original devd with some exceptions, a project page: project page
2) R4FBSDk prject page and FreeBSD Kernel Programming Interface and Kernel Module Interface
KPI is a FFI to kernel
KMI is kernel module standart library which used KPI and turns it into Rust friendly safe code. i.e like libc crate and nix crate.
+ cargo based build kernel module template a template of the kernel module without FreeBSD build system (best option)
+ makefile based kernel module template atemplate of the kernel module based on makefile which builds the module as a library and attaches it to C file (bad approach)

The reasons for termination are:
I have to focus my on my personal safety and security as I fleed from my country and I can not return.
I don't want to deal with projects based in USA anymore, sorry.
Also, threre are other reasons which are not really important for me, but also significant. One reason is some members FreeBSD community so arrogant that they believe that they are masters of memory management and they belittle my work by calling it "Rust sh@t". I, personally, don't care because I also doing this way quite often (even calling Rust this way). I am not big fan of Rust, but it is way better than C and C++ for sure. Provide better alternative and I will be happy to use it. And, I don't want to start a holywar, but it's not like that. In a large project it's very difficult to track and remember where the memory was allocated and how it was allocated. And I noticed that I am spending less time debugging the problems related to memory access violations and use after free problems or invalid pointers or out of bounds reading or even integer overflow.
And finally, I don't see any advantages of FreeBSD over GNU/Linux any more. Only two advantages still actual: there is no Linus Torvalds and FreeBSD is still optimal as it can.

If you have any question, you can ask, but I am not sure that I will reply.
Datz it.
 
Thank you for that information, and thank you for working on making FreeBSD better.

eesekaj said:
Recently I have released the devd(8) re-implemented in Rust (ported from closed source code which I developed for the client) and I was planning to submit port for review.
Click to expand...
The original code was developed for "a client", with whom you probably had an agreement that decides copyright and ownership of the code. The port is a derived work, and may be covered by the original copyright and ownership agreement. Are you sure this can be freely released?
 
ralphbsz said:
Thank you for that information, and thank you for working on making FreeBSD better.


The original code was developed for "a client", with whom you probably had an agreement that decides copyright and ownership of the code. The port is a derived work, and may be covered by the original copyright and ownership agreement. Are you sure this can be freely released?
Click to expand...

Forgive any oversteps but I noticed on the 'project page' (link above) the following (not sure if this is the information you were looking for ralphbsz):
<SNIP> Later then, a client has made a lot of changes in the code and I decided to release the code with modifications removing the proprietary code and replacing it with rewritten one. I have convinced the company for which the project was developped to allow me to release that code and now it is awailable. <SNIP>
Click to expand...

eesekaj You look to be a BSD expert to me, why wouldn't you stay on BSD?!
 
Because it's true - they are masters of memory management.

And yes, sometimes it makes sense to stand your ground, and sometimes it makes sense to just let things go...
 
Why is it always "memory management"? There are tons of severe security vulnerabilities around that are not related to that. This kind of discussion might give the dangerous idea just using e.g. Rust could be a "silver bullet" to achieve secure code. It is not.

When talking about FreeBSD base code, we're mostly talking about C, not C++. So, looking at C, how do you avoid memory errors? It's actually not that hard, following a few simple rules:
  • For every allocated object, define and use functions to allocate and free it, and use them in pairs.
  • Always initialize your objects on allocation. Explicitly initialize "uninitialized" fields with special values marking that (e.g. NULL for pointers).
  • Use expressions with sizeof, not type names (avoids wrong size calculations on refactoring).
  • Avoid "magic numbers" for array sizes, use constants (macros) or variables as appropriate.
  • Define a clear "owner" for any allocated object which is responsible for the lifecycle.
  • Where this is absolutely impossible, resort to transparent reference counting (create a function to take a reference which increments a reference counter, and in the function freeing the object, decrement the reference counter).
All of this of course requires a well designed and structured program. And as this is all manual work, of course you can still mess up with e.g. simple typos, but these are easy to spot with tooling available these days, like valgrind, like the sanitizers offered by the compiler, etc... I'd say the major remaining risk is programmers getting "clever", breaking their design e.g. for some kind of "optimization" and getting that wrong ... C certainly won't stop you.

Now guess what, severe security holes are very often a result of poor design/structure, in any language.

Getting back to Rust, actually this language transparently enforces some of the rules above, plus other things that help with robust programs like having objects immutable by default. Of course this is an advantage for writing "good" software. But it's certainly not a silver bullet. On the question whether Rust should be used in FreeBSD's base, if I'd look only at the language, I'd say "sure, why not", but my actual answer is still "no" for other reasons: I don't like that there's only a single implementation and no independent (standardized) language specification. For me, this raises concerns about durability of the code written in it. There's quite some C code written 20 to 30 years ago which still builds and runs fine these days without any modifications, and IMHO that's an important thing for an operating system. Will this be similar for Rust code?
 
zirias@ said:
Why is it always "memory management"?
Click to expand...
I also don't get it - especially if your promoted solution has 5 times the memory footprint... (in the given rdevd metrics 4024k vs 18296k !!!)

Also what on earth is going on with context switches in rust? 428 vs 12247?? thats a factor of 28!
Shouldn't be the goal to *improve* such vital code instead of making it *much* slower and inefficient (3.5x sys+user time...)? There's already enough horribly unoptimized code out there, justified by "compooters are so fast, we don't need no fast code/language", but this flawed mindset should never be introduced into the basic parts of the OS.

TBH my impression of all that "we have to use rust in base" discussion is, that it's just another incarnation of the "hip new language syndrome". Give it a few years and the same discussion will be started again with another new language (and C will still remain the better option).
 
sko said:
I also don't get it - especially if your promoted solution has 5 times the memory footprint... (in the given rdevd metrics 4024k vs 18296k !!!)
Click to expand...
Rust associates each object with an "owner" (i.e., bit of code). The compiler can (theoretically) track the lifetime of the owner to ensure the memory doesn't persist beyond its lifetime (e.g., a memory leak).
sko said:
TBH my impression of all that "we have to use rust in base" discussion is, that it's just another incarnation of the "hip new language syndrome". Give it a few years and the same discussion will be started again with another new language (and C will still remain the better option).
Click to expand...
+42

It is distressing how much code gets rewritten just because someone wants to play with some language du jour. Try explaining to your employer why he should add a dependency on such a "novelty" in his organization!

Why aren't we all using Ada, exclusively, in all projects?? <rolls eyes>

Why aren't scissors designed with 100 pound handles (to prevent folks from RUNNING with them)? Ans: treat their users as responsible people and teach them the consequences of running -- but give THEM the decision as to when to ignore that advice.

["My friend's sleeve is stuck in the car door and he is being dragged down the street. I need to get to him quickly to cut it off before he is seriously harmed!!"]
 
sko the most widespread argument for moving to Rust is "you get memory safety for free", and after sorting out the misconception that this would automatically ensure secure code (see above), your post is a nice example for the fact that nothing is really "for free".

I would still weaken your point a bit, because I think optimization should never be the first thought. What you absolutely must target is correctness, once you're sure enough about that, you can start thinking about efficiency etc. And there, the first question should be whether there's a real need for improvement.

For many parts of an operating system, I agree the answer will probably be "yes", because the code is executed often enough to affect the overall system performance of any user of that OS. Still some tools coming with the OS probably don't need the best possible performance. And there are a few more "but"s: We're looking at (presumably?) linear factors for efficiency here. In many cases, it's more important how performance scales with amount of input data, and this can only be "tuned" with better algorithms. Also, it's quite possible to improve on the factors you mention in the implementation of the language. In the end, the features of Rust compared to C of course can't be "for free", but they could most likely cost much less...

I'm getting back to what I personally consider most important for a language to become more than "yet another hype": An independent standard for the language specification. I remember articles in some "Amiga magazine" back then comparing the performance of code compiled with "Aztec C", "Lattice C" and then later, "GNU C", the first two were commercial products. Similar comparisons were done quite often between GCC and LLVM much later. As for a language targeting "systems programming", performance of the generated code is still a pretty important factor; I think some competition could trigger innovation (on the technical side, with a "fixed" language specification) for Rust as well.
 
zirias@ said:
For many parts of an operating system, I agree the answer will probably be "yes", because the code is executed often enough to affect the overall system performance of any user of that OS. Still some tools coming with the OS probably don't need the best possible performance. And there are a few more "but"s: We're looking at (presumably?) linear factors for efficiency here. In many cases, it's more important how performance scales with amount of input data, and this can only be "tuned" with better algorithms. Also, it's quite possible to improve on the factors you mention in the implementation of the language. In the end, the features of Rust compared to C of course can't be "for free", but they could most likely cost much less...
Click to expand...
I can't see how a compiler can "know" the performance characteristics of the targeted hardware in sufficient detail to be able to eke out the levels of performance that are necessary for a specific type of application (an OS being a very specialized application). E.g., how will the compiler know how frequently a particular piece of code executes (relative to some other piece of code) to be able to determine how best to pack structs to minimize cache line misses?

For example, a task's state is often encoded in a single struct -- because that's how humans think about the problem.

But, a more effective layout is to group similar fields from different tasks in contiguous memory so the algorithms that examine them can benefit from previous cache fills (e.g., when deciding which task to execute next).

The (hardware) industry's solution is to just keep providing bigger and bigger (and more layers) caches to try to compensate for poor locality of reference. The language designers seem to rely on the hardware to make up for inefficiencies in the way the language coerces the developer to think about their problems. And, developers assume every one ELSE is covering for their misgivings...
 
zirias@ said:
Why is it always "memory management"
Click to expand...
Hey good job everybody hijacking the thread and making it about the worst thing this person experienced! I'm sure you won't have the self realization necessary to be kind about this and just keep on going, instead of posting in the other thread or making a new one.

Anyway! So if they come back to read this, I'll save them the effort: The simple answer is that the computer can do the memory management for you and it becomes a non-existent bug class. Then you can focus on other types. One might argue that it's "never true" or "there will be bugs in unsafe" and I'm gonna say that it will be meaningfully true and that you can use your master of memory management skills in the unsafe to make it 100% true. I wish this guy luck in proving people wrong, because, TBH, things would be better if the people they perceived as haters were proven wrong. Since proof is doing work, godspeed.

Oh I forgot to address the "for free" part: Same argument assembler people made when the compiler came.

Whatever! The proof is in doing. Just let them do stuff and see if it works. I mean, if you asked me, people are just hating on it because they're scared it will work, convinces enough people (despite others thinking "the wrong tradeoffs were made"), and then they'd feel compelled to learn Rust, and they just don't wanna.

I mean, it's really hilarious that people think the speed of Rust vs C is a problem when most people out here using that speed to run... JavaScript.
 
msplsh said:
I mean, it's really hilarious that people think the speed of Rust vs C is a problem when most people out here using that speed to run... JavaScript.
Click to expand...
Actually, most folks are probably running ASM and C -- in their keyboards, mice, disk drive controllers, monitors, OS/drivers, etc.
But, you don't want to see that because it invalidates your argument. Let me know when you see JS (or rust) being used widely to design "appliances" (as they, almost certainly, are far more concerned about reliability and availability than a desktop OS (where the user can "reboot" and/or troubleshoot a misbehaving application).

When was the last time you rebooted your mouse? Keyboard? Disk drive(s)? Microwave oven? TV? Furnace? Thermostat? Any of the ~50+ processors in your car?

"I mean, it's really hilarious that people think the reliability of desktop software is a problem when most devices are NOT desktops!"
 
Don Y said:
I can't see how a compiler can "know" the performance characteristics of the targeted hardware in sufficient detail to be able to eke out the levels of performance that are necessary for a specific type of application (an OS being a very specialized application).
Click to expand...
If this was meant to contradict my points, I'll probably have to make explicit that what you quoted from me referred to the extra cost introduced by current Rust as outlined by sko and nothing else.

Don Y said:
For example, a task's state is often encoded in a single struct -- because that's how humans think about the problem.

But, a more effective layout is to group similar fields from different tasks in contiguous memory so the algorithms that examine them can benefit from previous cache fills (e.g., when deciding which task to execute next).
Click to expand...
This kind of optimization is manual work in C, introducing a technically motivated split in your objects (therefore severely affecting the design of your code, so, something you'll only do where you're sure you need it). It'll be the same in any other language. I wouldn't expect any compiler to do things like this automatically any time soon, that's not what I meant. 😉

edit: anecdotal side note, this kind of splitting is done "to the extreme" on the mos 6502, not because of cache locality (there is none), but because indexed addressing modes are limited to 256 bytes by 8bit index registers 😏
 
zirias@ said:
This kind of optimization is manual work in C, introducing a technically motivated split in your objects (therefore severely affecting the design of your code, so, something you'll only do where you're sure you need it). It'll be the same in any other language. I wouldn't expect any compiler to do things like this automatically any time soon, that's not what I meant
Click to expand...
My point is that all of the THOUSANDS of languages out there flail about with silly tweeks to syntax or development paradigms -- but don't address real performance, structure and maintenance issues.

I, as a developer, should be free to express my algorithm in a form that "makes sense to me" -- to minimize cognitive loading. But, a good language would be able to free itself from my "human" concerns and sort out how best to reorganize the code and the structs.

If I write a loop to clear (or otherwise "process") a section of memory a byte at a time, I wouldn't be surprised if the compiler generated code to do it in some wider storage unit to minimize the number of iterations and maximize available memory bandwidth. As long as the constraints on each side of that region were met, why should I care?

We don't object (welcome!) to compilers moving code around to gain a performance advantage. Or, eliminating steps that it deems as unnecessary. Why not let it sort out the whole application -- by providing the developer with mechanisms to express these "constraints" and letting the compiler have a free hand as long as those constraints aren't violated? Developers should spend their time creating MODELS of applications and letting compilers figure out how to implement them. Languages should be designed to facilitate this modeling; not micromanage the steps to implement a particular algorithm.

"Gee, you're looking at the scheduling parameters for the tasks in this tight loop -- but, only looking at the stored process/task state for ONE such task after you've exited the loop. Let's move the "state" portion of the structs into separate structs linked to these other parts of the struct..."
 
Don Y said:
We don't object (welcome!) to compilers moving code around to gain a performance advantage. Or, eliminating steps that it deems as unnecessary. Why not let it sort out the whole application -- by providing the developer with mechanisms to express these "constraints" and letting the compiler have a free hand as long as those constraints aren't violated? Developers should spend their time creating MODELS of applications and letting compilers figure out how to implement them. Languages should be designed to facilitate this modeling; not micromanage the steps to implement a particular algorithm.
Click to expand...
You know, these ideas aren't exactly new. In fact, my diploma thesis was in the area of model-driven development (and that's quite a while ago). Part of the complexity is that there are lots of these "constraints". And many are domain-specific...

Don Y said:
"Gee, you're looking at the scheduling parameters for the tasks in this tight loop -- but, only looking at the stored process/task state for ONE such task after you've exited the loop. Let's move the "state" portion of the structs into separate structs linked to these other parts of the struct..."
Click to expand...
The C language standard does an IMHO awesome job (especially remarkable considering its age) defining the "abstract machine" and the exact meaning of "observable behavior", giving compilers great freedoms with automatic optimizations. Still, C can't support what you outlined here. It gives some guarantees about the layout of data in memory in relation to how you declare your structs. Sometimes, that's important to allow the programmer to do things "right" concerning constraints (in the simplest case interfaces to code outside the current compilation unit) the compiler can't possibly know... 🤷
 
zirias@ said:
Why is it always "memory management"? There are tons of severe security vulnerabilities around that are not related to that. This kind of discussion might give the dangerous idea just using e.g. Rust could be a "silver bullet" to achieve secure code. It is not.
Click to expand...

Ok, this "silver bullet" argument nobody brought up is a "dangerous idea", but ...

zirias@ said:
When talking about FreeBSD base code, we're mostly talking about C, not C++. So, looking at C, how do you avoid memory errors? It's actually not that hard, following a few simple rules:
Click to expand...

... this is not? Seriously? :-/

Reality just doesn't work that way. Take FreeBSD src for example. As a reviewer I'm often confronted with overly long functions, switches, goto sections and monster loops where it's difficult to check all possible exit paths. And it's not that this code was written by amateurs without any sense of design and structure. C lacks some essential features to support safe programming (e.g. RAII), and the economics of C programming incentivize some bad programming constructs. It's not a shame to admit that, regardless of what one thinks about Rust - C was designed in a different time.

BTW, there's no significant difference in runtime performance between C, C++ or Rust. Once compiled, relative performance stays within one digit percentage, with no language a clear winner for all sorts of tasks. Libraries are a different topic though.
 
noise said:
BTW, there's no significant difference in runtime performance between C, C++ or Rust. Once compiled, relative performance stays within one digit percentage, with no language a clear winner for all sorts of tasks. Libraries are a different topic though.
Click to expand...

Totally unchecked C arrays will beat "proper" collection classes any day. It might stay within one percent for a whole bigger application as you say, though.
 
You must log in or register to reply here.
Back
Top