NodeJS modules creeping into ports

T

tomxor

I've noticed various node modules popping up as new port PRs, there are some where a port makes sense for better system integration e.g nodeBB... but others make little sense, I question the reasoning of needlessly duplicating modules from one package manager to a more system specific package manager without any clear advantages.

There are countless node modules with a CLI that can be used globally, from simple little tools to large web-development orientated task runners, front end package managers etc... grunt, gulp, brocolli, bower... the list goes on.

The standard way to use these is with npm install -g package-name. This works fine, why do we need them in ports? Is this common behaviour for other runtime package managers to leak into ports?
 
One of the reasons might be that a specific port depends on one or more of those modules. Ports can only be added when all their dependencies are met.
 
Ahh, ok. Now i look more closely some of them do have blocking PRs.

But that raises another question, is it really sustainable to pull in every dependency from the node world into the port world, wouldn't some kind of bridge between npm and pkg be better.
 
That's not going happen, pkg will not grow any extensions to handle packages of foreign package managers such as npm. The whole design of it is based on the principle that it handles its own packages only and whatever else is used on the system should be used in parallel with pkg, not as a dependency to pkg.
 
Maybe "bridge" was a poor choice of words, I don't think pkg needs (and can understand why it would not want) extensions...

All a port needs is node as a dependency then to be able to run npm install and npm update, this is little more than a one line script. But for limitations that I do not understand this doesn't appear to be possible as seen in this Thread 56659

Maybe I'm just not getting the isolated concept of ports. Seems like a lot of duplication.
 
tomxor said:
All a port needs to be able to do is run npm install and npm update, this is little more than a one line script.
Click to expand...
This might be easy to implement in the port, packages are a different matter.
 
Yeah and ports system is moving away constantly from such hackery and moving dynamic operations to pkg to be run at package install time instead of port build time. Ideally the port descriptions (the Makefiles now) wouldn't contain any program code and wouldn't be written in make(1) but in a descriptive language such as UCL (http://github.com/vstakhov/libucl).
 
tomxor said:
All a port needs is node as a dependency then to be able to run npm install and npm update, this is little more than a one line script. But for limitations that I do not understand this doesn't appear to be possible as seen in this Thread 56659
Click to expand...
TL;DR: Quality Assurance

The reasons for this are outlined in the Porter's Handbook https://www.freebsd.org/doc/en_US.ISO8859-1/books/porters-handbook/bundled-libs.html (the links to the Fedora/Gentoo Wikis go into a lot more detail). Just letting npm do its thing basically goes against almost all of these points.

The FreeBSD ports tree does not exist in a vacuum and other similar systems struggle with the same problem e.g. see this blog post about npm and Guix: https://web.archive.org/web/20160407141555/http://dustycloud.org/blog/javascript-packaging-dystopia/.

The same problem also exists with Haskell packages and Cabal. I'm currently building FreeBSD packages for http://haskellstack.org out of tree. I've a port ready for it on http://bugs.freebsd.org, but it bundles all dependent Haskell libraries because currently not all dependencies exist in the ports tree (or have the right versions) yet. I'm currently taking a variant of the "f... it, I'm out" approach from the blog post to building Stack.

Stack has ~ 136 (transitive) dependencies and I do not have a good overview if there are security vulnerabilities or critical bugs in any of them. And if they do have any it's my responsibility to rebundle all dependencies and bump the port revision to fix them.

kpa said:
Ideally the port descriptions (the Makefiles now) wouldn't contain any program code and wouldn't be written in make(1) but in a descriptive language such as UCL (http://github.com/vstakhov/libucl).
Click to expand...
Is someone working on this?
 
tobik said:
Is someone working on this?
Click to expand...

Not that I know of but the portmgr@ people have many times expressed dissatisfaction over the use of make(1) and Makefiles as a description language for ports because it's often way over the head of a typical port maintainer to fully understand make(1) and its pitfalls and this often results in bizarre bugs in ports that are hard to track.
 
tobik said:
...I've a port ready for it on http://bugs.freebsd.org, but it bundles all dependent Haskell libraries because currently not all dependencies exist in the ports tree (or have the right versions) yet. I'm currently taking a variant of the "f... it, I'm out" approach from the blog post to building Stack.
Click to expand...

That's what hukadan resorted to for nodeBB, and this might seem like a bit of a "f... it, I'm out" approach and overly static but If calling other package managers from pkg is a nono then I think it's actually better to bundle the whole node_modules tree with the package instead of attempting to duplicate dependant packages in ports. npm creates local trees for each module recursively (allowing for multiple conflicting versions within the same tree) so trying to jam this into a peer dependency style package system just seems like a bad idea.

tobik said:
Stack has ~ 136 (transitive) dependencies and I do not have a good overview if there are security vulnerabilities or critical bugs in any of them. And if they do have any it's my responsibility to rebundle all dependencies and bump the port revision to fix them.
Click to expand...

This was my concern too, pick a single big module like gulp and there will be thousands of node dependencies, they are evolving continuously, no port maintainer would ever be able to keep up unless it was automated... even small inconspicuous looking modules can have very large trees. Obviously npm naturally takes care of this and scrapes off the appropriate git revision of each repo directly.
 
Why not have a package that depends on a package in another package manager (ie npm), emit a message saying "hey install x y z through npm install -g x y z"?
 
node is all the rage. As is web dev. So all the cool kids want to use npm and node. It's like, ya' know, totally rad down in the valley so like, ya' know?

And tomorrow I have to tell a client, for the first time in ages, we're going to miss Friday's deadline because of node, callback hell, and anything related to npm gives me the dry heaves.
 
Generally speaking, I think anything that winds up in a foreign package repository should be handled by a foreign "package manager." That package manager can be made available in a FreeBSD port/package, and just end there. If these things are done right (Ruby gems, Cabal sandboxes), then the environment is self-contained. This is the sensible way to do things, because there's no way to possibly handle everything that might go along with a given language-specific package or distribution.

As an example, Jekyll is a very popular static site generator. It's available in the ports tree. So are a handful of Jekyll packages. But Jekyll has several official add-ons, dozens of third-party plugins, a whole bunch of extra glue people have hacked together, and so on. So if you really want to make the most of it, you need to manually install the Ruby gem using devel/ruby-gemsrather than www/rubygem-jekyll. If you use the --user-install flag, the gems get installed to ~/.gem. They never touch the rest of your system, you can manage them on your own, and you can easily scrap the whole directory and start over if need be. This makes a lot more sense than expecting a half-dozen people maintaining different ports to crack the whip at each other every time something needs fixing. Of course, this is just me assuming that npm is capable of a self-contained installation environment...
 
Python2 and Python3 have a similar package and ports distribution on FreeBSD. Some key modules like tkinter and sqlite3 are available as separate ports/packages for several Python versions. I think either they're a dependency for other ports/packages or it was simply safe (and easy) to build them, because all of the underlying dependencies (and there are many) could be satisfied.

Can the same not be done for NodeJS? Make ports of only the modules that need to exist separately?
 
ANOKNUSA said:
... the gems get installed to ~/.gem. They never touch the rest of your system, you can manage them on your own, and you can easily scrap the whole directory and start over if need be. This makes a lot more sense than expecting a half-dozen people maintaining different ports to crack the whip at each other every time something needs fixing. Of course, this is just me assuming that npm is capable of a self-contained installation environment...
Click to expand...

npm config set prefix "~/.local"

(For just a user) or system wide with the --global flag. You still need to add prefix path + "/bin" to your shell path.

drhowarddrfine said:
<node rant>
Click to expand...

nodejs is awesome and you are wrong... or nodejs sucks and you are right... or nodejs exists but is new and has some horrible nastly parts to it that some people (me) have to deal with.

It's a new kind of flamewar all it's own :p but this thread is for discussing how it's modules should or should not be separated from FreeBSD's ports.
 
tomxor said:
nodejs exists but is new and has some horrible nastly parts to it that some people (me) have to deal with.
Click to expand...
I have to deal with it on a daily basis.

tomxor said:
this thread is for discussing how it's modules should or should not be separated from FreeBSD's ports.
Click to expand...
Exactly why I brought it up. As mentioned above, and by many articles throughout the net, node and npm can be a hacked together mess maintained by multiple, unrelated, uncoordinated people where packages fail constantly. I only use it when I have to and I only have to cause one API, some of our clients use, only supplies a node interface and that is where we spend all our waking hours.
 
Not to go to far OT... but I find I can reduce the pain by carefully choosing (or not choosing) modules. The vast majority of node modules do not do one thing well. Most do either way more or way less than one thing (yes less, like one line of code that is useless on it's own), often within the same massive tree, and often the same modules multiple times.

So if i'm writing a nodejs module I don't consider my dependencies as individual... I consider their whole tree and decide whether it's too large and too stupid and too much of a liability to be worth it's utility. You can argue this all stems from nodejs not having something like a standard library, it's also because people thoughtlessly depend on things, made possible by the nested dependency resolution.
 
tomxor And those are all the reasons not to use npm and node for this (hence, why it's on topic). Makefiles are great (stated without knowing or understanding the problems of the port maintainers).
 
That is a good point... if the solution was to use npm to take care of port dependencies for pkg then deployment could get pretty flaky.

As i said before, if invoking npm from pkg is not acceptable then I think it would be better to call it while building the port and bundle the node_modules tree with the package... they are supposed to be isolated little trees not peer dependencies, so why don't we treat them that way and not attempt to flatten them into ports... which i think could get crazy pretty quickly.

[EDIT]

I think this discussion is getting too abstract, speculative and preemptive (all my fault).

I went and had a look at the gruntjs PR. It doesn't build for whatever reason, but if I understand the it correctly he's trying to use an npm shrinkwraped dependency tree to capture it at a specific state (all version locked), which makes it reliable...

It's a really good idea, you could put a shrinkwrapped package.json in a port which depends on only node, then fetch all the repos and recreate the tree (cos we aren't allowed to use npm to fetch).

However I'm suggesting this should be done directly for a non-node port that relies on node modules rather than arbitrarily pouring top level node modules into ports... unless it is desired to use it directly e.g nodeBB.
 
You must log in or register to reply here.
Back
Top