Living on the edge

[rbranco]

I'm using -CURRENT on a VM where I can do snapshots in case something breaks, etc.

I want to know the experience of people using -CURRENT for serious workloads.

Why do you use CURRENT?

How many times it broke?

Do ZFS snapshots help in this case? Which other tools and processes you use to make sure nothing breaks?

 

[YYY__]

It allows you to look into the future. That also includes new hardware drivers. Here is a list of things from Netflix. It's not my case. Sorry for going off track.

Enjoy Netflix? You Should Thank FreeBSD

Netflix is one of the most popular streaming services in the world. But you already know that. Don’t you? What you probably did not know is that Netflix uses FreeBSD to deliver its content to you. Yes, that’s right. Netflix relies on FreeBSD to build its in-house content

itsfoss.com

Here are the benefits Netflix sees of tracking FreeBSD:

• Quicker feature iteration
• Quicker access to new FreeBSD features
• Quicker bug fixes
• Enables collaboration
• Minimizes merge conflicts
• Amortizes merge “cost”

 

[cracauer@]

I always run -current. Occasionally you have to repair X11 (DRM module or whatever). I like to see unwanted changes as soon as possible. Keep in mind it isn't my main desktop.

The last time I had to roll back a FreeBSD install was a botched 14-release update from source. The last heavy bug I got scared of was the ZFS corruption bug which made it into all stable versions of FreeBSD, Solaris and Linux.

The biggest drawback of running -current is frequent ABI changes, which means you can no longer install new packages until you do a `make world`. Which recommends a reboot.

 

[Cath O'Deray]

❝… Some people run -STABLE in production. Heck, some people run -CURRENT in production. Different risk appetites…❞

— Colin Percival, Lead, Primary Release Engineering Team, December 2023
{link removed}​

 

[Cath O'Deray]

<https://forums.freebsd.org/search/361791/?c[users]=rbranco&o=date> for me to gauge where you're coming from.

I think of FreeBSD-CURRENT as leading edge (not bleeding edge).

Yesterday:

If you make good use of ZFS boot environments, you should have no fear.

A few hours ago in Discord we walked someone through upgrading from 14.0-RELEASE to 15.0-CURRENT. Partly with reference to the wiki, which currently focuses on RELEASE, so some variation was required (plus, we noticed some gaps).

Production use, for me. CURRENT has been my everyday system (daily driver), on notebooks, for at least eight years. From my profile here:

… switched from OS X 10.9.5 Mavericks to PC-BSD. Then TrueOS, then FreeBSD-CURRENT with KDE Plasma. …

Context: <{link removed}>, where TrueOS used FreeBSD-CURRENT from the outset. (I did use PC-BSD long before the transition, so I probably used 9.2-CURRENT for a while.)

… How many times it broke? …

For me, it's fair to say never. A Startpage.com search of a freebsd-current archive finds just one post from me, part of a July 2023 thread:

• {link removed}

(Off-topic: there may be something wrong with Google's index of FreeBSD's archives. This does not alter the essence of how I feel about CURRENT breaking.)

I have encountered issues whilst using CURRENT, however it's very rare for me to find an issue with CURRENT itself.

I can't recall anything other than occasional build failures. Build failures can be a thing of the past – since pkgbase allows the operating system, and packages of ports, to be updated with ease:

pkg upgrade

 

[cracauer@]

I had a machine hang on 15-current yesterday when playing with segfault handlers (userland). Might have been a panic, but X11 was up so I wouldn't know. Can't reproduce.

 

[indomitableMoron]

I'm using -CURRENT on a VM where I can do snapshots in case something breaks, etc.

I want to know the experience of people using -CURRENT for serious workloads.

Why do you use CURRENT?

How many times it broke?

Do ZFS snapshots help in this case? Which other tools and processes you use to make sure nothing breaks?

I'm running a mix of 13.2 and 14. No CURRENT. Too chicken.

 

[cracauer@]

Intel DRM kernel modules from packages also don't load today.

 

[Cath O'Deray]

… The biggest drawback of running -current is frequent ABI changes, which means you can no longer install new packages until you do a `make world`. …

I never found installation impossible.

Intel DRM kernel modules from packages also don't load today.

Which version?

uname -aKU

x11/nvidia-driver-470 here, so I'm quite complacent about kernel modules in ports. Following a pkgbase update to CURRENT:

root@mowa219-gjp4-zbook-freebsd:~ # uname -aKU
FreeBSD mowa219-gjp4-zbook-freebsd 15.0-CURRENT FreeBSD 15.0-CURRENT main-n268036-d682c1eaa598 GENERIC-NODEBUG amd64 1500012
 1500012
root@mowa219-gjp4-zbook-freebsd:~ #

 

[Cath O'Deray]

… Might have been a panic, but X11 was up so I wouldn't know. …

I have preferences set for an automated restart, so if a long panic occurs with X11 frozen, waiting long enough will lead to a relatively short dump and then files in /var/crash following the restart.

 

[Cath O'Deray]

Three pkgbase updates to base over the weekend, the first of which led to long panics. Most noticeable, consistently, very soon after Plasma appeared with restored windows (photograph attached). Nothing obviously relevant on the freebsd-current list for the period <{link removed}>.

Soon enough, I found an almost entirely reliable workaround, continued using the bugged builds for the rest of the weekend. After creation of a thread in Discord, bsdimp there dropped a hint about reviews in Phabricator.

Back to work today, so (for peace of mind) I'm simply using the boot environment that preceded 62d47d73b7eb.

From Discord:

<{link removed}>

Spoiler: Recent boot environments (notes to self)

root@mowa219-gjp4-zbook-freebsd:~ # bectl list -c creation | tail -n 10
n267824-0dd5a5603e7a-b -      -              530M  2024-01-31 09:25
n267824-0dd5a5603e7a-c -      -              409M  2024-02-04 19:48
1500012-a              -      /tmp/1500012-a 50.1M 2024-02-06 09:32
1500012-b              -      /tmp/1500012-b 122M  2024-02-06 22:31
1500012-c              -      /tmp/1500012-c 2.27G 2024-02-08 01:07
1500013-a              -      /tmp/1500013-a 2.25G 2024-02-08 23:31
1500013-b              NR     /              242G  2024-02-10 07:44
1500013-c              -      /tmp/1500013-c 1.78G 2024-02-10 14:11
1500013-d              -      /tmp/1500013-d 1.97G 2024-02-11 04:59
1500013-e              -      /tmp/1500013-e 5.15G 2024-02-11 13:29
root@mowa219-gjp4-zbook-freebsd:~ # strings /tmp/1500012-a/boot/kernel/kernel | egrep ^FreeBSD\ 15.0
FreeBSD 15.0-CURRENT main-n268036-d682c1eaa598 GENERIC
root@mowa219-gjp4-zbook-freebsd:~ # strings /tmp/1500012-a/usr/include/sys/param.h | grep 15000
#define __FreeBSD_version 1500012
root@mowa219-gjp4-zbook-freebsd:~ # strings /tmp/1500012-b/boot/kernel/kernel | egrep ^FreeBSD\ 15.0
FreeBSD 15.0-CURRENT main-n268036-d682c1eaa598 GENERIC
root@mowa219-gjp4-zbook-freebsd:~ # strings /tmp/1500012-b/usr/include/sys/param.h | grep 15000
#define __FreeBSD_version 1500012
root@mowa219-gjp4-zbook-freebsd:~ # strings /tmp/1500012-c/boot/kernel/kernel | egrep ^FreeBSD\ 15.0
FreeBSD 15.0-CURRENT main-n268079-e4ab361e5394 GENERIC
root@mowa219-gjp4-zbook-freebsd:~ # strings /tmp/1500012-c/usr/include/sys/param.h | grep 15000
#define __FreeBSD_version 1500012
root@mowa219-gjp4-zbook-freebsd:~ # strings /tmp/1500013-a/boot/kernel/kernel | egrep ^FreeBSD\ 15.0
FreeBSD 15.0-CURRENT main-n268104-d04abb05375d GENERIC
root@mowa219-gjp4-zbook-freebsd:~ # strings /tmp/1500013-a/usr/include/sys/param.h | grep 15000
#define __FreeBSD_version 1500013
root@mowa219-gjp4-zbook-freebsd:~ # strings /boot/kernel/kernel | egrep ^FreeBSD\ 15.0
FreeBSD 15.0-CURRENT main-n268149-eb86c6c5b462 GENERIC
root@mowa219-gjp4-zbook-freebsd:~ # strings /usr/include/sys/param.h | grep 15000
#define __FreeBSD_version 1500013
root@mowa219-gjp4-zbook-freebsd:~ # uname -aKU
FreeBSD mowa219-gjp4-zbook-freebsd 15.0-CURRENT FreeBSD 15.0-CURRENT main-n268149-eb86c6c5b462 GENERIC amd64 1500013 1500013
root@mowa219-gjp4-zbook-freebsd:~ # strings /tmp/1500013-c/boot/kernel/kernel | egrep ^FreeBSD\ 15.0
FreeBSD 15.0-CURRENT main-n268154-62d47d73b7eb GENERIC
root@mowa219-gjp4-zbook-freebsd:~ # strings /tmp/1500013-d/boot/kernel/kernel | egrep ^FreeBSD\ 15.0
FreeBSD 15.0-CURRENT main-n268164-c0b8047bdc13 GENERIC
root@mowa219-gjp4-zbook-freebsd:~ # strings /tmp/1500013-e/boot/kernel/kernel | egrep ^FreeBSD\ 15.0
FreeBSD 15.0-CURRENT main-n268174-5f7ac491eef4 GENERIC
root@mowa219-gjp4-zbook-freebsd:~ # bectl umount 1500012-a && bectl umount 1500012-b && bectl umount 1500012-c && bectl umount 1500013-a
root@mowa219-gjp4-zbook-freebsd:~ # bectl umount 1500013-c && bectl umount 1500013-d && bectl umount 1500013-e
root@mowa219-gjp4-zbook-freebsd:~ # bectl list -c creation | tail -n 10
n267824-0dd5a5603e7a-b -      -          530M  2024-01-31 09:25
n267824-0dd5a5603e7a-c -      -          409M  2024-02-04 19:48
1500012-a              -      -          50.1M 2024-02-06 09:32
1500012-b              -      -          122M  2024-02-06 22:31
1500012-c              -      -          2.27G 2024-02-08 01:07
1500013-a              -      -          2.25G 2024-02-08 23:31
1500013-b              NR     /          242G  2024-02-10 07:44
1500013-c              -      -          1.78G 2024-02-10 14:11
1500013-d              -      -          1.97G 2024-02-11 04:59
1500013-e              -      -          5.15G 2024-02-11 13:29
root@mowa219-gjp4-zbook-freebsd:~ #

 

[Cath O'Deray]

FreeBSD CURRENT stabilization cycle​

… In the last decade quality of FreeBSD CURRENT improved so much, that not only brave developers run it on their laptops, but also large companies use it. Time to bring it to a new level. Every individual or a business that use CURRENT has their own protocol of how to stay up date and avoid disasters. An individual will first update their desktop and only after that will update server(s). A company would run their internal regression test suite or some other validation protocol. Right now we all do that independently from each other having little coordination and providing little help each other. We also do not broadcast to the world that FreeBSD CURRENT is usable. I've seen a lot of people who stay away from CURRENT based on their 20-year old experience with it. …

— glebius@, 2024-02-24
also archived (with clickable links) at <{link removed}>​

Related:

Please create a -current forum

Looks like folks would like to have a place to discuss experiences with the -current branch. Better here than on Reddit. kpedersen convinced me that this is a bad idea here: https://forums.FreeBSD.org/threads/please-create-a-current-forum.79595/post-503475

forums.freebsd.org

 

[Cath O'Deray]

x11/kde5 currently not packaged for FreeBSD:15:amd64 <{link removed}>, I'm using pkg to upgrade base but not ports until it reappears.

Postscript: the package became available a few hours later.

 

[Cath O'Deray]

… panics. Most noticeable, consistently, very soon after Plasma appeared …

For anyone who's curious:

• the panics did not relate to graphics, or Plasma
• my workaround was Wi-Fi (iwm(4)) instead of wired networking (em(4))
• the wait for a fix was not long.

 

[Cath O'Deray]

Attached: a transcript of an upgrade from 1500014 to 1500015 – using pkgbase.

NB: repository names are not base and FreeBSD. Both names are too vague.

In retrospect, the bfs commands – seeking .pkgsave files – were probably unnecessary. See the comment from bapt@ at <{link removed}>.

Re: the long list of boot environments, I previously used letters of the alphabet (in names of environments) to help signify order. Now, instead, I'm using numbers:

• 1500015-01-base represents my first ugprade to base that resulted in version 1500015 of the OS.

A condensed list of commands from the transcript:

1. pkg update -r FreeBSD-base && pkg update -r FreeBSD-ports ; date
2. bectl list -c creation
3. bectl create 1500015-01-base
4. bectl mount 1500015-01-base /tmp/up
5. time pkg -r /tmp/up upgrade --yes --quiet --repository FreeBSD-base --fetch-only
6. time pkg -r /tmp/up upgrade --yes --quiet --repository FreeBSD-base --no-repo-update && grep pkg /var/log/messages | tail -n 1
7. grep 20758 /var/log/messages
8. cp /tmp/up/boot/loader.efi /boot/efi/efi/freebsd/loader.efi
9. du -hs /tmp/up/var/cache/pkg
10. pkg -r /tmp/up clean -a --quiet --yes && pkg -r /tmp/up autoremove
11. pkg update -r FreeBSD-base && pkg update -r FreeBSD-ports ; date
12. history -S
13. cp /root/.history /tmp/up/root/.history
14. bectl umount 1500015-01-base
15. bectl activate 1500015-01-base ; exit

 

[Cath O'Deray]

March 2024 stabilization week began yesterday …

… runs up to Friday 18:00 UTC, but if there is consensus that any regressions discovered by participants have been fixed, it will end early. …

 

[astyle]

I'm using -CURRENT on a VM where I can do snapshots in case something breaks, etc.

I want to know the experience of people using -CURRENT for serious workloads.

Why do you use CURRENT?

How many times it broke?

Do ZFS snapshots help in this case? Which other tools and processes you use to make sure nothing breaks?

I'm running 15-CURRENT in a VBox VM on a 13.2-RELEASE host... My reason for that - I'm trying to compile my way into a KDE6 desktop. Nothing broke so far, but I should probably give the VM more than 8 GB of RAM (host has 40 GB). And the VM does have a serious workload - I already have failed rust and gcc compilations where I had to step in and install packages from latest for those failed compilations. Those failed compilations ran overnight, too.

Playing with ZFS snapshots is probably next on the agenda for that VM.

 

[Cath O'Deray]

Thanks,

… trying to compile my way into a KDE6 desktop. …

How do your options differ from what's already available as packages?

(In a nutshell.)

 

[astyle]

Thanks,



How do your options differ from what's already available as packages?

(In a nutshell.)

Well, in a nutshell, I try to turn on EVERYTHING in a port when I compile it. Packages usually come with a pretty conservative set of options turned on. For example, I turn on all of the audio options available (SNDIO, Pipewire, PulseAudio, JACK, and ALSA...), because they compile no problem, and it's a way to avoid weird lack of sound later on. I also turn on LTO, and turn off the EXAMPLES options.

 

[zirias@]

Why do you use CURRENT?

I did use 11-CURRENT for testing on my desktop and notebook, because 10-RELEASE missed GPU drivers.

Since 11-RELEASE, I only use -CURRENT in VMs dedicated to testing ports with poudriere (with jails for -CURRENT and all currently supported -RELEASE versions). Ports explicitly support -CURRENT, so you can't skip testing with it for serious work on ports.

How many times it broke?

11-CURRENT kernel-panicked a few times for me ...

None of my port testing VMs ever had issues so far (except for the expectable occassional ABI-breakage like when base OpenSSL was upgraded to 3.x, which is easily fixable by rebuilding some ports locally).

 

[Cath O'Deray]

pkg upgrade

Of around two hundred boot environments, since I switched to pkgbase four months ago, AFAICT only three were non-bootable. An outline:

% grep 15000 ./boot\ environments.txt | head -n 1
1500012-a                       2024-02-06 09:32 main-
% grep 15000 ./boot\ environments.txt | wc -l
     199
% grep not\ bootable ./boot\ environments.txt
1500018-77-base                 2024-05-26 04:22 not bootable
1500019-006-base-ports          2024-06-05 23:02 main-n270596-dbe7ff254e6c not bootable
1500019-007-base                2024-06-06 04:44 main-n270620-9e012a9da5df not bootable
% tail -n 4 ./boot\ environments.txt
1500019-019-base                2024-06-15 07:31 main-n270777-ad9cc86bf60c
1500019-020-base-linux-c7       2024-06-15 21:36 main-n270777-ad9cc86bf60c
1500019-021-base-rl9            2024-06-16 06:44 main-n270779-d0cb4674df97
1500019-022-base-ports          2024-06-17 14:56 main-n270823-7deadea2eb0a
%

Living on the edge? Maybe.

Boot failure? Not a problem, I simply restart the OS

---

I can't properly count how many of the 199 environments did not involve base. From what's below, I guess: less than thirty.

% grep 15000 ./boot\ environments.txt | grep ports | grep -v base | wc -l
       9
%

 

[Cath O'Deray]

… boot does not proceed. No splash screen, and so on. …

Have you encountered problems with CURRENT generally?

Nope.

The failure above was no problem, because I made the boot environment only temporarily active before the boot.

root@mowa219-gjp4-zbook-freebsd:~ # history | grep 'bectl activate' | tail -n 3
  1990  14:28   bectl activate -t 1500025-013-base
  2010  17:02   history | grep 'bectl activate'
  2011  17:02   history | grep 'bectl activate' | tail -n 3
root@mowa219-gjp4-zbook-freebsd:~ #

(I very rarely risk activating without a prior temporary activation.)

The next boot was fine – 1500025-012-base – I used bectl to mount 1500025-013-base at /tmp/up, overwrote my custom splash screen with a scaled-down edition, unmounted, then again tested 1500025-013-base.

---

I could easily have caused the same failure with customisation in stable/14 … there'll be the same opportunity to shoot oneself in the foot with releng/14.2 in November.