Call for Foundation-supported Project Ideas

[mark_j]

Accept Olivier Certner's Tauthon port.

It's already a port, or do you mean changing all python2 references/uses to use this port? (I know nothing about Tauthon).

 

[BSD-Kitsune]

For those here, I did not mean to make things go OT -- I simply was replying to someone's chatter on my comments. It got outta hand fast.

I'm out. Good luck guys.

 

[Cath O'Deray]

… There is a thread …

Thanks, and for reference:
<{link removed link provider grahamperrin is dead}> | <{link removed}> | <{link removed}>

 

[Jose]

For those here, I did not mean to make things go OT -- I simply was replying to someone's chatter on my comments. It got outta hand fast.

I'm out. Good luck guys.

Well, I think a new file system generally, and porting XFS specifically are very much on-topic.

I'm very interested in this suggestion because the lack of zero-copy sendfile(2) in ZFS makes it unsuitable for some high performance applications. It's also my understanding ZFS does not play nice with Postgresql. The Freebsd workaround is to use UFS2. Maybe having XFS as an option would improve the suitability of Freebsd for these applications.

 

[Jose]

It's already a port, or do you mean changing all python2 references/uses to use this port? (I know nothing about Tauthon).

That port was in the tree for less than 24 hours before it was removed in what has been described as a rage commit. Mr. Certner had added it specifically to work around the Python 2 dependency in the Palemoon build. He even went to the trouble of getting the obstreperous Palemoon upstream to agree they were ok with this workaround. He got treated very poorly for his trouble.

I get the feeling there are contentious discussions about the Chromium port in particular and the Python 2 build dependency in general that are being conducted behind closed doors. All we see is the fallout which is frankly embarrassing for the project. "No Python 2 in the tree!!!1! All these ports with build dependencies on it will be removed post-haste! Please don't look over at the Chromium port."

 

[Cath O'Deray]

PostgreSQL

… my understanding ZFS does not play nice …

July 2021, with reference to a 2017 article:

… According to the well known Postgres consultant company 2nd Quadrant ZFS and Postgres do perform really well with ZFS' COW features being turned on. So snaphots and all the advanced stuff. …

Also from 2017: {link removed} (thanks to Eric A. Borisch for {link removed}).

 

[Cath O'Deray]

Chromium

Upstream – Issue 942720: Make our code work with Python 3 – various comments followed a {link removed} but (sorry) I lack the knowledge to decipher the technical aspects.

A blocker – Issue 1251670: Make sure test_runner and utils for iOS are py3 compatible/converted to py3 – was updated nine days ago. Essentially (with added emphasis):

… starting to plan wrap up for the chromium/src Py3 migration, actually, we are targeting to stop supporting Py2 by EO2022Q1, but we cannot do that until all test systems are done. …

– in plain English, targeting end of the first quarter of 2022.

I get the feeling there are contentious discussions about the Chromium port in particular and the Python 2 build dependency in general that are being conducted behind closed doors. All we see is the fallout …

I'm blissfully unaware of fallout probably because I have not looked where I should not but (like many people, I guess) understanding of the reasons, not least Chromium, for lang/python2 remaining in the ports tree for so long after its end of life.

 

[Alain De Vos]

That why i dont run kde/plasma/chromium.

PS1, my tuning for postgresql is the following (feel free to comment):

recordsize 8K
checksum skein
compression lz4
atime off
primarycache all
logbias throughput

PS2, filosofical question, regarding UFS are all these features really needed or can't some be dropped,
-background fsck
-fsck pruning
-snapshots

 

[bakul]

Well, I think a new file system generally, and porting XFS specifically are very much on-topic.

When SGI open sourced XFS, they put it under GPL so it won’t become part of any BSD licensed OS.

What should be done, but is likely very difficult, is a better integration of zfs code and virtual memory and caching so that extra copies are avoided.

 

[Jose]

PostgreSQL



July 2021, with reference to a 2017 article:



Also from 2017: PostgreSQL + ZFS – Best Practices and Standard Procedures (thanks to Eric A. Borisch for the pointer).

Those are generic ZFS + Postgresql howtos. There are significant problems with ZFS (and other COW filesystems) in write-intensive applications. See slide 39 here:

PostgreSQL on EXT4, XFS, BTRFS and ZFS

PostgreSQL on EXT4, XFS, BTRFS and ZFS - Download as a PDF or view online for free

www.slideshare.net

Some googling around leads me to believe XFS is the performance leader for Postgresql, but a lot of the benchmarks out there are suspect because they use ZFS without tuning it for Postgresql's idiosyncrasies.

 

[Jose]

When SGI open sourced XFS, they put it under GPL so it won’t become part of any BSD licensed OS.

XFS is licensed GPL-2 and is in the Linux kernel so it will likely not ever change to GPL-3. The latter is incompatible with the BSD license, but the former is not. Witness the large chunks of Linux kernel source in graphics/drm-kmod.

 

[Alain De Vos]

Postgresql Note: The more you tune for performance the more data you lose on a power outage. Because stuff is longer held in memory and not synced to disk.
So you can not blindly look to numbers.
[& Primary cache metadata is not always a good idea, eg when database data does not fit in memory or for large selects]

 

[bakul]

XFS can at best be a port but who will do the work? I suspect the foundation will prioritize funding work that directly goes into FreeBSD. The work is likely to be fairly involved but may be if interested people can raise enough money to support it, it can happen!

 

[Cath O'Deray]

… UFS …background fsck …

IIRC background checking is appropriate where the file system is tuned for soft updates but not soft update journaling.

 

[Cath O'Deray]

… a lot of the benchmarks out there are suspect because they use ZFS without tuning it for Postgresql's idiosyncrasies. …

FWIW the one paragraph at <{link removed}> although (again, sorry) that might be too generic.

Thanks for the SlideShare link. There's a slightly more recent (2016) set with a TL;DR: Use ZFS, even in the cloud, however it was a lightning talk so maybe not suitably in depth.

 

[Cath O'Deray]

lang/python2

I don't get why this is still in ports. Just get rid of this old junk. All ports which depend on it can go too.

… www/qt5-webengine …

FreeBSD bug {link removed}

• depends on {link removed} and
• blocks {link removed}

 

[Alain De Vos]

XFS can at best be a port but who will do the work? I suspect the foundation will prioritize funding work that directly goes into FreeBSD. The work is likely to be fairly involved but may be if interested people can raise enough money to support it, it can happen!

There is (only) read-write userspace xfs via fusefs-lkl

 

[BSD-Kitsune]

When SGI open sourced XFS, they put it under GPL so it won’t become part of any BSD licensed OS.

What should be done, but is likely very difficult, is a better integration of zfs code and virtual memory and caching so that extra copies are avoided.

XFS can be reimplemented under another license.

XFS can at best be a port but who will do the work? I suspect the foundation will prioritize funding work that directly goes into FreeBSD. The work is likely to be fairly involved but may be if interested people can raise enough money to support it, it can happen!

It's a nontrivial task for sure. I'd be open to another filesystem, but ZFS is unsuitable as a main filesystem, and UFS2 is simply deficient.

 

[Beastie7]

I said it's pretty "ehh' at best. doesn't mean terrible.

The bones of it are good... ish. But the soft-updates system has failed. Soft-updates lost out to journaling and nowadays I/O performance from more modern filesystems is better. We don't need to force everyone who wants decent performance onto ZFS.

In particular, though, I can name three entrenched issues:

1. It's based on 1970s-era technology with all the same deficiencies and short-sighted aspects of it. UFS had a benefit years ago when all the BSDs were relatively speaking the same dialect. Nowadays, none of them really understand each other as each has gone in a different, incompatible direction. The 1970s were nearly 50 years ago, and documentation for the FS is not as good as some better ones.

2. XFS in particular is extremely well documented, enough that making an independent reimplementation of directory structure v2 era XFS is basically quite possible and then merging newer changes in shouldn't be terrible.

3. The amount of work to correctly document UFS2, get it speaking even the OpenBSD dialect (impossible mostly because OpenBSD's is a pure-soft updates version) and get it up to modern performance standards is more work than porting a new filesystem.

4. Even if you did all that it would lack many of the better aspects of say XFS, Reiser4 or other filesystems.

This says absolutely nothing. Exactly how, and what areas of storage management is UFS lacking in? I'd like to see technical details and benchmarks too. Are you familiar with the GEOM framework in FreeBSD? UFS has been trailing, if not, exceeding Linux's options. (mainly due to GEOM). If your main use case is the desktop or embedded, UFS is perfectly fine. If it's enterprise storage; ZFS is second to none obviously.

 

[BSD-Kitsune]

This says absolutely nothing. Exactly how, and what areas of storage management is UFS lacking in? I'd like to see technical details and benchmarks too. Are you familiar with the GEOM framework in FreeBSD? UFS has been trailing, if not, exceeding Linux's options. (mainly due to GEOM). If your main use case is the desktop or embedded, UFS is perfectly fine. If it's enterprise storage; ZFS is second to none obviously.

ZFS is bloated. I like it, I use it, but it's bloated and not suitable for all workloads.

Can DragonFlyBSD's HAMMER Compete With Btrfs, ZFS? - Phoronix

www.phoronix.com

UFS vs. ZFS File-System Performance On FreeBSD 9.0 - Phoronix

www.phoronix.com

FreeBSD's UFS vs Ext4

muc.lists.freebsd.questions.narkive.com

I once emailed Larabel on why he doesn't test UFS anymore, and it's because performance-wise it always trails to the point it isn't worth setting up test systems.

If you're in denial about UFS2's deficiencies performance-wise, maybe the fact it's not well-documented compared to say, XFS, will convince you.

Every interface of XFS is documented, and its codebase is easy to understand with lots of comments and easy to understand functionality.

XFS Overview & Internals

XFS Training Course

web.archive.org

Wayback Machine

Here's some examples of SGI-derived documentation.

There is no whitepaper on FreeBSD's UFS2, let alone any modern implementation. There's no mutual-intelligibility between the different BSDs UFS2 versions, let alone with the System V UFS. The codebase is ancient, and has been limped along into the modern era. There's been no promises of fixing UFS2. There's been no promises of improving its performance. People in the community will say "Well ackshually if you want good performance you should use ZFS." and that's a non answer. That's admitting that a filesystem that requires paravirtualizing parts of the OpenSolaris kernel has a massive footprint is superior to your native, default filesystem.

Ask yourself: Is that a good look?

A small team could easily implement XFS from scratch using documentation alone. I am just one person. If there's such a team in the future, sign me up.

 

[astyle]

OK, this looks like a good opportunity for me to chime in... to answer jrm@ 's opening post, I have a few items on my wishlist:

1. Get AMD GPU programming going under FreeBSD. OpenCL would be nice, but even better - get the HIP API going.
2. Clean up KDE upgrading in ports, as opposed to pkg. Right now, I'm kludging things together with Poudriere and portsnap, and I'm writing tons of little helper scripts that use find, grep, sed, awk, and other utilities to go through lists and files. Fun project that I'm doing for myself, but I'm hoping there's something in the official project that makes this kind of chore easier. I want to upgrade just KDE (KF5 and Plasma), compile fresh tarballs against an existing base. In the past, upgrading KDE with pkg would mean pulling in strange deps that I would not set if I prepared my own packages. And the procedure always came with the risk of messing up my entire system, forcing a clean re-install.
3. Start using Calamares installer for the system?

 

[hardworkingnewbie]

It's not a new topic: for FreeBSD if you want to use Postgres for more than just tinkering around, it's either ZFS or begone.

There was even 2020 a talk at the BSDCAN about that phenomenon:

https://papers.freebsd.org/2020/BSDcan/munro-PostgreSQL_on_FreeBSD.files/munro-PostgreSQL_on_FreeBSD.pdf

One issue with UFS from that talk:
UFS doesn’t recognise this access pattern as sequential. Linux does, due to read-ahead “window”. ZFS apparently does too.

 

[Alain De Vos]

An easy way to improve postgresql performance is to change the postgresql tunables:

fsync=
synchronous_commit=
wal_sync_method=

In that case, after a power outage database inconsistency & data-loss will be inevitable.

 

[Cath O'Deray]

… to answer [FONT=monospace]jrm@[/FONT] 's opening post, …

Detailed discussion of XFS might continue here:

XFS on BSD

There's also <https://forums.freebsd.org/tags/xfs/>, and these two XFS-specific topics (although neither is tagged as such), both solved: XFS support (2017–2021), Where can I find the status of XFS for FreeBSD (2021).

… OpenCL would be nice, …

astyle's topic on the subject, for anyone who'd like to continue there:

Using OpenCL on FreeBSD

I'm trying to get OpenCL going on FreeBSD. The GPU that I have is an Asus Radeon RX 550 4 GB. Per these instructions, I installed lang/clover, benchmarks/clpeak, devel/ocl-icd, deve/clinfo, and graphics/drm-kmod. I successfully got a Plasma Wayland session going, so I know graphics/drm-kmod is...

forums.freebsd.org

Clean up KDE upgrading in ports, as opposed to pkg.

XenForo can not find {link removed} the required topic(s). astyle please, do you have a link handy? I'm almost certain that we have prior discussion.

 

[astyle]

I don't think I ever had a complete discussion of cleaning up the KDE upgrading... I used a Google search (https://www.google.com/search?q=KDE+astyle+site:forums.freebsd.org) to see what in the world I could have been saying on the topic. I discovered that I mostly made random comments promoting usage of ports over pkg for installing KDE and setting options in makefiles:

Akonadi won't start

The error from akonadictl start --verbose is org.kde.pim.akonadiserver: Failed to connect to database! org.kde.pim.akonadiserver: Database error: "Can't connect to local MySQL server through socket '/var/run/user/10 01/akonadi/mysql.socket' (2) QMYSQL: Unable to connect" is the database...

forums.freebsd.org

or

Wasting time on ...

My waste time was sometime trying to get good "use-flags" for Gentoo-linux. (side-node it is impossible, because they always clash, you can never get them right). What was your waste time , a technical impossible task ?

forums.freebsd.org

or

Shell - Getting a handle on sh syntax

I'd like for some help with getting a handle on sh syntax... The goal of my rather simple script is to generate a list of all installed ports whose category is kde or kde-applications... To this end, I have the following in list_kde_ports.sh: for port in `cat port_list.txt` do if [ -x pkg query...

forums.freebsd.org

The last one is the closest I was able to find on having a complete conversation... But even this one was just me getting some help to get over a hurdle (making lists of ports that I need to compile with Poudriere), I'm nowhere close to achieving what I said I want to do with Poudriere in post #46