When to write userland vs kernel code?

toddg

toddg

I am writing a jail orchestration system that includes a software defined network (SDN). I've written this about 4x now, the current version works but is not satisfactory (see this last post). I'm thinking of ditching config files entirely, and coding directly to the FreeBSD APIs...

Here's the outline:

Bash: 
Orchestrator (gleam/erlang) -> [something] -> FreeBSD APIs

FreeBSD APIs:
* networking (ifconfig)
* firewall (pf)
* containers (jails | oci (podman))
* load balancer (currently Caddy(userland))
* file system (zfs)
* packages (pkg)

Questions:

* What is the decision tree for when to write userland code vs a loadable kernel module (KLD)?
* What abstraction libraries exist? For instance, I saw a video recently where the author of PF mentioned a wrapper library that folks should use instead of the raw PF APIs...
* BTW - Has anyone tried implementing something like Caddy load balancer as a KLD?

Note: [cbsd](https://github.com/cbsd/cbsd) is close to what I want to build. But it doesn't have an SDN, and it's written in C...

Resources:
* Freebsd Device Drivers (Kong)
* https://github.com/ebrandi/FDD-book
 
My general rule is to always write for userland, unless you actually need to work in the kernel. Although I'm not a freebsd developer. Perhaps prototype the thing in user space, then consider whether some of it would be better implemented in kernel once you have a prototype; or use the prototyping effort to identify which parts can only be implemented as kernel modules. Apart from anything else, this approach gives better portability to other operating systems.

Of course in this case, it may well be that kernel implementation is the best way, for some of it at least. It sounds like that kind of a project.
 
Kernel level code may be required when high performance and low latency are required or you must access some privileged instructions or resource. Kernel code is generally harder to debug & any security bug in it would be harder to fix and more catastrophic. Writing fault tolerant kernel code will be much harder as well. Almost always user mode code will suffice. Even when you must implement something in kernel code, keep it minimum.

For what you describe, kernel mode doesn't seem necessary.

toddg said:
Note: [cbsd](https://github.com/cbsd/cbsd) is close to what I want to build. But it doesn't have an SDN, and it's written in C...
Click to expand...
Consider using it first. Collaborating with them to provide some added functionality you want may be far easier than trying to implement something from scratch. Also check out existing ports.
 
Use kernel mode if any of the following are true
1) you need direct access to the hardware, bypassing device driver APIs
2) you need access to the real memory pages of any or all other processes
3) you have tight realtime requirements that cannot be handled in user mode with realtime scheduling class
4) you need to manages user-mode processes from a layer deeper in the onion shell model

Otherwise stick with userland programming
 
THe answer is not flat rate. You need to look at the individual pieces separately. I assume that your master program/controller will run in userland, so you would also have to provide an interface to userland if you do your things in kernel modules. But back to the individual pieces.

First is ifconfig

I you look at the ifconfig sources you see that it does not read kernel memory to gets it information but uses ioctl and other interfaces. That's good for your use case. ifconfig does not yet have a -libxo output option to json, which would be good for you to use. But alas it is not there.

What I would do to manage interfaces is use netlink. FreeBSD added a Linux compatible netlink mechanism to the kernel. That is better than random ioctls. It's a suitable interface to userland.

 
Thank you cracauer@ . I'll follow your suggestion ad read the ifconfig sources and then look into netlink.

Lot's of great advice in this thread, thx everyone!
 
You must log in or register to reply here.
Back
Top