Call for Foundation-supported Project Ideas

BSD-Kitsune

Active Member

Reaction score: 71
Messages: 168

For those here, I did not mean to make things go OT -- I simply was replying to someone's chatter on my comments. It got outta hand fast.

I'm out. Good luck guys.
 

Jose

Daemon

Reaction score: 1,158
Messages: 1,371

For those here, I did not mean to make things go OT -- I simply was replying to someone's chatter on my comments. It got outta hand fast.

I'm out. Good luck guys.
Well, I think a new file system generally, and porting XFS specifically are very much on-topic.

I'm very interested in this suggestion because the lack of zero-copy sendfile(2) in ZFS makes it unsuitable for some high performance applications. It's also my understanding ZFS does not play nice with Postgresql. The Freebsd workaround is to use UFS2. Maybe having XFS as an option would improve the suitability of Freebsd for these applications.
 

Jose

Daemon

Reaction score: 1,158
Messages: 1,371

It's already a port, or do you mean changing all python2 references/uses to use this port? (I know nothing about Tauthon).
That port was in the tree for less than 24 hours before it was removed in what has been described as a rage commit. Mr. Certner had added it specifically to work around the Python 2 dependency in the Palemoon build. He even went to the trouble of getting the obstreperous Palemoon upstream to agree they were ok with this workaround. He got treated very poorly for his trouble.

I get the feeling there are contentious discussions about the Chromium port in particular and the Python 2 build dependency in general that are being conducted behind closed doors. All we see is the fallout which is frankly embarrassing for the project. "No Python 2 in the tree!!!1! All these ports with build dependencies on it will be removed post-haste! Please don't look over at the Chromium port."
 

grahamperrin

Son of Beastie

Reaction score: 1,015
Messages: 3,432

grahamperrin

Son of Beastie

Reaction score: 1,015
Messages: 3,432

Chromium

Upstream – Issue 942720: Make our code work with Python 3 – various comments followed a July 2021 progress report but (sorry) I lack the knowledge to decipher the technical aspects.

A blocker – Issue 1251670: Make sure test_runner and utils for iOS are py3 compatible/converted to py3 – was updated nine days ago. Essentially (with added emphasis):

… starting to plan wrap up for the chromium/src Py3 migration, actually, we are targeting to stop supporting Py2 by EO2022Q1, but we cannot do that until all test systems are done. …

– in plain English, targeting end of the first quarter of 2022.

I get the feeling there are contentious discussions about the Chromium port in particular and the Python 2 build dependency in general that are being conducted behind closed doors. All we see is the fallout …

I'm blissfully unaware of fallout probably because I have not looked where I should not :) but (like many people, I guess) understanding of the reasons, not least Chromium, for lang/python2/ remaining in the ports tree for so long after its end of life.
 

Alain De Vos

Son of Beastie

Reaction score: 869
Messages: 2,826

That why i dont run kde/plasma/chromium.

PS1, my tuning for postgresql is the following (feel free to comment):
Code:
recordsize 8K
checksum skein
compression lz4
atime off
primarycache all
logbias throughput

PS2, filosofical question, regarding UFS are all these features really needed or can't some be dropped,
-background fsck
-fsck pruning
-snapshots
 

bakul

Active Member

Reaction score: 92
Messages: 164

Well, I think a new file system generally, and porting XFS specifically are very much on-topic.
When SGI open sourced XFS, they put it under GPL so it won’t become part of any BSD licensed OS.

What should be done, but is likely very difficult, is a better integration of zfs code and virtual memory and caching so that extra copies are avoided.
 

Jose

Daemon

Reaction score: 1,158
Messages: 1,371

PostgreSQL



July 2021, with reference to a 2017 article:



Also from 2017: PostgreSQL + ZFS – Best Practices and Standard Procedures (thanks to Eric A. Borisch for the pointer).
Those are generic ZFS + Postgresql howtos. There are significant problems with ZFS (and other COW filesystems) in write-intensive applications. See slide 39 here:
Some googling around leads me to believe XFS is the performance leader for Postgresql, but a lot of the benchmarks out there are suspect because they use ZFS without tuning it for Postgresql's idiosyncrasies.
 

Jose

Daemon

Reaction score: 1,158
Messages: 1,371

When SGI open sourced XFS, they put it under GPL so it won’t become part of any BSD licensed OS.
XFS is licensed GPL-2 and is in the Linux kernel so it will likely not ever change to GPL-3. The latter is incompatible with the BSD license, but the former is not. Witness the large chunks of Linux kernel source in graphics/drm-kmod.
 

Alain De Vos

Son of Beastie

Reaction score: 869
Messages: 2,826

Postgresql Note: The more you tune for performance the more data you lose on a power outage. Because stuff is longer held in memory and not synced to disk.
So you can not blindly look to numbers.
[& Primary cache metadata is not always a good idea, eg when database data does not fit in memory or for large selects]
 

bakul

Active Member

Reaction score: 92
Messages: 164

XFS can at best be a port but who will do the work? I suspect the foundation will prioritize funding work that directly goes into FreeBSD. The work is likely to be fairly involved but may be if interested people can raise enough money to support it, it can happen!
 

grahamperrin

Son of Beastie

Reaction score: 1,015
Messages: 3,432

… a lot of the benchmarks out there are suspect because they use ZFS without tuning it for Postgresql's idiosyncrasies. …

FWIW the one paragraph at <https://openzfs.github.io/openzfs-docs/Performance and Tuning/Workload Tuning.html#postgresql> although (again, sorry) that might be too generic.

Thanks for the SlideShare link. There's a slightly more recent (2016) set with a TL;DR: Use ZFS, even in the cloud, however it was a lightning talk so maybe not suitably in depth.
 

grahamperrin

Son of Beastie

Reaction score: 1,015
Messages: 3,432

Alain De Vos

Son of Beastie

Reaction score: 869
Messages: 2,826

XFS can at best be a port but who will do the work? I suspect the foundation will prioritize funding work that directly goes into FreeBSD. The work is likely to be fairly involved but may be if interested people can raise enough money to support it, it can happen!
There is (only) read-write userspace xfs via fusefs-lkl
 

BSD-Kitsune

Active Member

Reaction score: 71
Messages: 168

When SGI open sourced XFS, they put it under GPL so it won’t become part of any BSD licensed OS.

What should be done, but is likely very difficult, is a better integration of zfs code and virtual memory and caching so that extra copies are avoided.

XFS can be reimplemented under another license.

XFS can at best be a port but who will do the work? I suspect the foundation will prioritize funding work that directly goes into FreeBSD. The work is likely to be fairly involved but may be if interested people can raise enough money to support it, it can happen!

It's a nontrivial task for sure. I'd be open to another filesystem, but ZFS is unsuitable as a main filesystem, and UFS2 is simply deficient.
 

Beastie7

Aspiring Daemon

Reaction score: 651
Messages: 738

I said it's pretty "ehh' at best. doesn't mean terrible.

The bones of it are good... ish. But the soft-updates system has failed. Soft-updates lost out to journaling and nowadays I/O performance from more modern filesystems is better. We don't need to force everyone who wants decent performance onto ZFS.

In particular, though, I can name three entrenched issues:

1. It's based on 1970s-era technology with all the same deficiencies and short-sighted aspects of it. UFS had a benefit years ago when all the BSDs were relatively speaking the same dialect. Nowadays, none of them really understand each other as each has gone in a different, incompatible direction. The 1970s were nearly 50 years ago, and documentation for the FS is not as good as some better ones.

2. XFS in particular is extremely well documented, enough that making an independent reimplementation of directory structure v2 era XFS is basically quite possible and then merging newer changes in shouldn't be terrible.

3. The amount of work to correctly document UFS2, get it speaking even the OpenBSD dialect (impossible mostly because OpenBSD's is a pure-soft updates version) and get it up to modern performance standards is more work than porting a new filesystem.

4. Even if you did all that it would lack many of the better aspects of say XFS, Reiser4 or other filesystems.

This says absolutely nothing. Exactly how, and what areas of storage management is UFS lacking in? I'd like to see technical details and benchmarks too. Are you familiar with the GEOM framework in FreeBSD? UFS has been trailing, if not, exceeding Linux's options. (mainly due to GEOM). If your main use case is the desktop or embedded, UFS is perfectly fine. If it's enterprise storage; ZFS is second to none obviously.
 

BSD-Kitsune

Active Member

Reaction score: 71
Messages: 168

This says absolutely nothing. Exactly how, and what areas of storage management is UFS lacking in? I'd like to see technical details and benchmarks too. Are you familiar with the GEOM framework in FreeBSD? UFS has been trailing, if not, exceeding Linux's options. (mainly due to GEOM). If your main use case is the desktop or embedded, UFS is perfectly fine. If it's enterprise storage; ZFS is second to none obviously.
ZFS is bloated. I like it, I use it, but it's bloated and not suitable for all workloads.


I once emailed Larabel on why he doesn't test UFS anymore, and it's because performance-wise it always trails to the point it isn't worth setting up test systems.

If you're in denial about UFS2's deficiencies performance-wise, maybe the fact it's not well-documented compared to say, XFS, will convince you.

Every interface of XFS is documented, and its codebase is easy to understand with lots of comments and easy to understand functionality.


Here's some examples of SGI-derived documentation.

There is no whitepaper on FreeBSD's UFS2, let alone any modern implementation. There's no mutual-intelligibility between the different BSDs UFS2 versions, let alone with the System V UFS. The codebase is ancient, and has been limped along into the modern era. There's been no promises of fixing UFS2. There's been no promises of improving its performance. People in the community will say "Well ackshually if you want good performance you should use ZFS." and that's a non answer. That's admitting that a filesystem that requires paravirtualizing parts of the OpenSolaris kernel has a massive footprint is superior to your native, default filesystem.

Ask yourself: Is that a good look?

A small team could easily implement XFS from scratch using documentation alone. I am just one person. If there's such a team in the future, sign me up. :)
 

astyle

Daemon

Reaction score: 756
Messages: 1,634

OK, this looks like a good opportunity for me to chime in... to answer jrm@ 's opening post, I have a few items on my wishlist:

1. Get AMD GPU programming going under FreeBSD. OpenCL would be nice, but even better - get the HIP API going.
2. Clean up KDE upgrading in ports, as opposed to pkg. Right now, I'm kludging things together with Poudriere and portsnap, and I'm writing tons of little helper scripts that use find, grep, sed, awk, and other utilities to go through lists and files. Fun project that I'm doing for myself, but I'm hoping there's something in the official project that makes this kind of chore easier. I want to upgrade just KDE (KF5 and Plasma), compile fresh tarballs against an existing base. In the past, upgrading KDE with pkg would mean pulling in strange deps that I would not set if I prepared my own packages. And the procedure always came with the risk of messing up my entire system, forcing a clean re-install.
3. Start using Calamares installer for the system? 😅
 

hardworkingnewbie

Well-Known Member

Reaction score: 287
Messages: 277

Alain De Vos

Son of Beastie

Reaction score: 869
Messages: 2,826

An easy way to improve postgresql performance is to change the postgresql tunables:
Code:
fsync=
synchronous_commit=
wal_sync_method=
In that case, after a power outage database inconsistency & data-loss will be inevitable.
 

grahamperrin

Son of Beastie

Reaction score: 1,015
Messages: 3,432

… to answer [FONT=monospace]jrm@[/FONT] 's opening post, …

👍

Detailed discussion of XFS might continue here:

XFS on BSD

There's also <https://forums.freebsd.org/tags/xfs/>, and these two XFS-specific topics (although neither is tagged as such), both solved: XFS support (2017–2021), Where can I find the status of XFS for FreeBSD (2021).

… OpenCL would be nice, …

astyle's topic on the subject, for anyone who'd like to continue there:


Clean up KDE upgrading in ports, as opposed to pkg.

XenForo can not find the required topic(s). astyle please, do you have a link handy? I'm almost certain that we have prior discussion.
 

astyle

Daemon

Reaction score: 756
Messages: 1,634

I don't think I ever had a complete discussion of cleaning up the KDE upgrading... I used a Google search (https://www.google.com/search?q=KDE+astyle+site:forums.freebsd.org) to see what in the world I could have been saying on the topic. I discovered that I mostly made random comments promoting usage of ports over pkg for installing KDE and setting options in makefiles:
or
or
The last one is the closest I was able to find on having a complete conversation... But even this one was just me getting some help to get over a hurdle (making lists of ports that I need to compile with Poudriere), I'm nowhere close to achieving what I said I want to do with Poudriere in post #46
 
Top