Python The sad state of Python on Freebsd

ralphbsz said:
And since the versions of some software in the container changes, this has to trigger a full test cycle. Oh wait, with CI/CD, we no longer test software.

I hope you can hear the sarcasm in the above statement.
Click to expand...
The Python container should only some basic libraries, Python, and that's it.

Ideally all software should have unit tests, functional & integration tests.
 
Hi
Why is there two versions of devel/py-setuptools-scm7 AND devel/py-setuptools-scm ( version 8 ) that is used in turn by different ports ?
Both cant be installed simultaneously .

depending on specific port I am building they request one or the other of these setup tools.
shouldn't Port maintainers request at least version 7 insted of port failing if version 8 is installed. ?

//Lars
 
ralphbsz said:
It's not just FreeBSD. I see some of the same problems on Linux too. And on my personal Mac, the problems only don't appear because ... will be explained below.

The root of the problem is that there are multiple systems used to install "packages". One is pkg, which is by definition OS-specific (only works on FreeBSD, duh). The other is pip, which is by definition language specific (only does python, duh). So even in a simpler world of running ONLY ONE operating system (FreeBSD) and ONLY ONE language (python), it gets painful:
  1. For some packages, you can only use "pkg install", because they need FreeBSD-specific patches, and the generic pip version does not work.
  2. For some packages, you can only use pip, because nobody has bothered to make a FreeBSD pkg.
  3. And for some, you can choose either package installed, but you don't get the same version, usually because the FreeBSD pkg maintainer (a human volunteer) is running behind.
  4. And for some packages, it works perfect.
Now when it comes time to do upgrades, you need to run "pkg update/upgrade", and the pip magic required to do all upgrades (which is not quite trivial, you first need to list the installed packages). And every time you do this, some packages may migrate back and forth among the four categories described above.

If you add virtual environments to the list, it gets even harder, because those 4 categories are now environment specific.

What is the fix? Tough question. Try to do updates, if all your applications work, go drink. If they don't, mess around (usually I move from pkg to pip, and that often helps), or go back to installing from source. I have exactly the same issues on Linux (Debian running on RPi in my case). And there it's even harder than on FreeBSD, because there is no way to do a major version upgrade sanely, so I regularly have to wipe and reinstall, which increases the churn between the four categories above. The only reason this doesn't affect me on the Mac is: There I use no package mechanism other than pip for Python code, because on the Mac there don't seem to be any OS-specific patches needed.

In your case, it's even worse, since you have to drag a Linux install into the mix too.
Click to expand...
pkg also works on many SystemV UNIX OSes (Solaris, OpenIndiana, ...) In fact, Solaris uses pkg as the default.
 
kpedersen said:
Using containers as a solution to that specific problem is basically admitting defeat that poor engineering is taking place!
  • Reusable libraries are great
  • Dependencies are bad

Unfortunately they are the same thing. It is important for the developer to make a good decision and not drag in, i.e the whole of NumPy just to interface with some contiguous C data (I see this one a lot).
Click to expand...
Jython .... all in one programing language, but unfortunately not default supported on any system, and the intense overhead.
 
rbranco said:
In the case of Python containers, if you pin the specific versions with `pip freeze` you're guaranteed to have the same Python dependencies.
Click to expand...
This just incurs technical debt. Sooner or later one of your pinned dependencies will have a catastrophic security failure and your tech debt will be foreclosed.
 
Jose said:
This just incurs technical debt. Sooner or later one of your pinned dependencies will have a catastrophic security failure and your tech debt will be foreclosed.
Click to expand...
You can use `~=` instead of `==` so it fetchs the latest minor version: django~=5.1.6

But it depends on the maintainers of those dependencies respecting semantic versioning, just as it depends on you to test and update yours.

The biggest security issue with downloading from PyPI (and this is not limited to just Python) is downloading a trojaned version that steals your data. That's where containers come in handy as they provide some isolation. They're not a magic bullet, but it's better than having them access root or home.
 
rbranco said:
You can use `~=` instead of `==` so it fetchs the latest minor version: django~=5.1.6
Click to expand...
That's worse. You still get breakage and dependency hell from projects whose developers don't understand semantic versioning, which is most of them.

Not that the the behavior you can expect from projects that do understand it is any better. You'll hang out in some minor version forever, blissfully unaware that upstream has deprecated it and moved on to the new shiny. At some point someone will find a vulnerability in your old and deprecated version. That's the day (and maybe the night) you get to try and update it under pressure with some CISO or similar lizard breathing down your neck.
 
Jose said:
That's worse. You still get breakage and dependency hell from projects whose developers don't understand semantic versioning, which is most of them. Not that the the behavior you can expect from projects that do understand it is any better. You'll hang out in some minor version forever, blissfully unaware that upstream has deprecated it and moved on to the new shiny. At some point someone will find a vulnerability in your old and deprecated version. That's the day (and maybe the night) you get to try and update it under pressure with some CISO or similar lizard breathing down your neck.
Click to expand...

And that's why all software must have unit tests, functional tests, integration tests and so on.

I never had a problem with vulnerable dependencies since I enabled automated security scanning in my Github projects. I'm notified everytime there's a CVE for a dependency. I think Gitlab & Codeberg have them too. You can scan OCI containers with trivy and other tools.
 
rbranco said:
And that's why all software must have unit tests, functional tests, integration tests and so on.

I never had a problem with vulnerable dependencies since I enabled automated security scanning in my Github projects. I'm notified everytime there's a CVE for a dependency. I think Gitlab & Codeberg have them too. You can scan OCI containers with trivy and other tools.
Click to expand...
Not an exact art but some research suggests ~150% improvement of finding vulnerabilities from the wholesale security scanning currently in use.

That still leaves a *lot* of vulnerabilities. My personal experience always tends to lean towards reducing dependencies as the better approach. (Preferably both of course).

Dependency aggregation languages like Python, Rust, Perl, NodeJS, simply require more ongoing maintenance because of this.

That said, what we are currently calling "AI" in 2025 is a set of algorithms that lead themselves very well to a static analysis use-case, so I am optimistic that the general irresponsible use of dependencies I see (particularly in the web ecosystem) is going to have some amount of damage control applied.

Doorknob said:
Jython .... all in one programing language, but unfortunately not default supported on any system, and the intense overhead.
Click to expand...
The monolithic "big standard library" approach by Java(+Jython), .NET, etc is remarkably good. But when things break, they really break. Imagine if Oracle one day decided that they didn't like java.net.Socket anymore and replaced it. The entire world would collapse (I see this a lot more with .NET where the fallout luckily tends to stay localized to medium-sized businesses).

With a more distributed approach (but not quite as distributed as the Python/Rust micro-dependencies) you get some more resilience. In general, I do honestly like the C/C++ middle ground of smaller standard library + POSIX. It stands the test of time really well.
 
kpedersen said:
Not an exact art but some research suggests ~150% improvement of finding vulnerabilities from the wholesale security scanning currently in use.

That still leaves a *lot* of vulnerabilities. My personal experience always tends to lean towards reducing dependencies as the better approach. (Preferably both of course).

Dependency aggregation languages like Python, Rust, Perl, NodeJS, simply require more ongoing maintenance because of this.
Click to expand...

Python packages are a PITA to maintain and even large Linux distros lack enough manpower for this.

OCI containers solve the packaging problem (of dependencies and needing root to install packages) and also provide some isolation.

I see a LOT of potential for FreeBSD here because Jails are the proper primitives to build containers unlike namespaces+cgroups as used by Linux than need seccomp+SELinux/AppArmor for hardening.

Linux offers way too much flexibility and this makes it hard to test all possible scenarios and has an expanded attack surface. FreeBSD can disrupt the whole thing via technological leapfrogging on lots of areas explored by Linux, and the containers area is one, thanks to podman based on Jails. FreeBSD only needs to build some critical mass by improving WiFi and other stuff on laptops, and sudden mass adoption will happen.
 
Hello! I was actually in the middle of building the latest Python release from source on FreeBSD 14.2, I just tested a basic Python file with Python version 3.13.2
pythonbsdtest.png
 

Attachments

  • bsd-python3.13.2-test.png
    bsd-python3.13.2-test.png
    92.3 KB · Views: 34
Just getting around to more Python tests. Currently you can build the latest Python release from source. But I recommend isolation to a Python virtual environment to prevent system-wide breakages on FreeBSD.
 

Attachments

  • pythonbenchmarks.png
    pythonbenchmarks.png
    176.4 KB · Views: 35
ok i may have to agree slightly. I cannot install Python libraries with pip. Some of these pip pkg packages are tailored for Python 3.11. Interesting
 

Attachments

  • pip.png
    pip.png
    159 KB · Views: 29
John Cleese has turned into a bit of a right wing nut job in his old age

he had a show on GB News ( which is a british fox news lite )
which also had presenters like Nigel Farage and Jacob Rees Mog
 
You must log in or register to reply here.
Back
Top