Synth: Introducing new custom package repository builder for FreeBSD and DragonFly

[wblock@]

I disagree it's semantics. Making a package is a specific thing with specific meaning. A package is not being made. The installation occurs directly from stage directory.

Case in point: if a package is being made, where is it?

Counterpoint: if a package is not being made, how can pkg-list(8) and pkg-delete(8) work on it? This is why I say it's semantics.

For the purposes of that doc, the files in the stage directory are equivalent to a package, in the same way that a coffee cup is topologically equivalent to a donut.

Here is what it says now:

When a port is compiled, it actually creates a package. The package is then installed and managed by the package management system.

I can make it more pedantic:

When a port is compiled, it creates the same files as would be contained in a single binary package file. Rather than compress those files into a single .txz binary package and then re-extract them, it just installs them directly. The package is then installed and managed by the package management system.

The second adds implementation detail which is irrelevant to the point and actually obscures it, that being that ports and packages are parts of the same system.

Again, I'm willing to reword this if there is a more technically accurate way to say it that does not muddy the point.

 

[marino]

I think the distinction is important because pkgsrc does create an actual package, so anyone familiar with it would have been misled by original wording.

I think the new wording isn't great either. This would be fine to me:

The system compiles the port and installs it in a staging area, and pkg(8) then directly installs it on to the system, similar to how it would install a package.

something like that.

 

[talsamon]

Ok, in the moment looks good (till the next update? )
Have removed rodent (sniff) - has a bug. Make PR's for editors/abiword and x11-fm/rodent.
With just-build no package was removed. I installed all 41 new (10 packages caused by me - changed audio options).
Changed options for graphics/blender to nearly standard (turned off cycles and openimageio - both broken)..
I don't know if graphics/blender caused the chaos. But synth installed before graphics/openimageio which is marked in the graphics/blender port as broken (synth - the tool for hobby-forensics).
After all I know as much before (except just-build).

everything that uses the removed packages are removed

I don't think this is not necessairly. The highest level port should check the next "deeper" level, but not more. Except the next deeper level will be also upgradet. The recursive check is an endless circle. It will be too much to compile, as I mentioned already above.

Postponed lang/ghc till tomorrow and the audio/jack problem to any of the next days. The box needs cool down, and I had after four days for the moment enough from compiling.

But very thanks for your patience.

 

[fernandel]

Did you update your ports tree? If so, that would be normal.
(or does your ports tree get updated with a cron job?)

Yes, I did . Thank you. Now I have a problem just with math/atlas (skipped ports).

 

[talsamon]

math/atlas seems broken (there exists a newer version but till now, nobody has update it), if you need for math/py-numppy take the option netlib or openblas.

 

[marino]

Yes, I did . Thank you. Now I have a problem just with math/atlas (skipped ports).

Try modifying /usr/ports/math/atlas/Makefile: Comment out the line starting with "MANUAL_PACKAGE_BUILD=" (put "#" in front of it). I would think it would build then.

 

[fernandel]

Try modifying /usr/ports/math/atlas/Makefile: Comment out the line starting with "MANUAL_PACKAGE_BUILD=" (put "#" in front of it). I would think it would build then.

Yes, it works. It's building 'skipped" ports. Thanks for the help.
It is done.

 

[PacketMan]

So far so good, its going through the list one by one, but I do have one issue: if I roll my mouse wheel it causes a graceful shutdown. Arrgggg did it a couple times already. Other than that so far so good.

For my own education would be nice on your webpage to have a thorough explanation of what each build status means, maybe you can put that in the man page. Also would be nice if the Built counter was split into two: Built-C (for ports/packages compiled locally on the machine), and Built-F (for packages fetched and extracted). Something for you to consider if you find yourself bored.

Its too early to tell but overall progress seems to be considerably faster than pure postmaster compile for all. I'm estimating 10 hours at this point in time.

Thanks again.

 

[Chris_H]

PacketMan
Regarding the moune-wheel; is your box strained? Meaning, are the resources pretty maxed out, while SYNTH is building?
Also; are you running SYNTH from your Desktop? I'm not marino@ , and I didn't develop SYNTH, but I knowing what I do, I can't imagine how SYNTH could be at all responsable, aside from further taxing an already fairly taxed system.

Just my thoughts. Hoping they might help.

--Chris

EDIT:
I just realized I mis-read your statement regarding the mouse-wheel; I somehow saw the shutdown as system shutdown -- my bad. Sorry.

 

[talsamon]

lang/ghc: failed depency check

========================< phase : package  >========================
===>  Building package for ghc-7.10.2
actual-package-depends: dependency on /lib/libutil.so.9 not registered (normal if it belongs to base)

.

reminds me on:
x11-fm/rodent

========================< phase : package  >========================
===>  Building package for ghc-7.10.2
actual-package-depends: dependency on /lib/libmagic.so not registered (normal if it belongs to base)

but the rodent error also appears in the port, I don't if it is also with lang/ghc.
(By the way it compiles with poudriere - with the error message).

Both files exists.

 

[Chris_H]

marino@ ,
My standard procedure for updating my production servers, has been to use traditional jails on the development box I just updated with ports-mgmt/synth. Given my experience, aside from a couple of (not unexpected) huckups with ports-mgmt/synth was so good. I got to thinking how I might use synth as an update strategy for updating my servers. A couple of thoughts come to mind. On one hand, I could simply point synth to a different set of Ports, PortsConf, and Packages folders, using copies of those I already use for them within the jails I currently use to update them. But given the servers are on different different versions, I'm thinking -- especially where Linux related ports are concerned, that I might run into issues. So I was thinking it might be a better idea, if I ran synth within a jail itself. Seems fiesable, but thought I'd like to get your thoughts on the matter. Are either of my ideas better than the other, or niether, or...?

Thanks!

--Chris

 

[Chris_H]

talsamon

(normal if it belongs to base)

As I read this; it is my understanding that if either of those libraries (libutil.so.9, or libmagic.so) come from the "base install" of FreeBSD, than you should expect to see those warnings. Meaning, that since synth builds the ports in a chroot(8) file system -- a jail(8) so to speak. Than the port builders do not have access to those libraries. TL;DR -- in short; it's OK. You do not have to be concerned.

All the best.

--Chris

 

[talsamon]

Than the port builders do not have access to those libraries

.. and it fails, and tries to compile again and again.

I edit above: it compiles/installs with poudriere and in the port - with the error message (rodent).

 

[Chris_H]

it compiles/installs with poudriere and in the port - with the error message (rodent).

If I'm not mistaken; ports-mgmt/poudriere does have access to those libraries. It may be that ports-mgmt/synth might need to better accomodate that situation. I'll defer to marino@ for all the technicalities, regarding that.

--Chris

 

[talsamon]

Yes, I understood. Maybe synth better detect the errors. My critc is that I could not exclude the failures, to prevent the tries to compile again and again. Mark broken is an ugly solution.

 

[marino]

talsamon Chris_H: Those aren't even errors. It's just information. All it is saying is that those libraries aren't registered in pkg(8) database and the most likely reason is that they are not PORTS libraries but BASE libraries. And actually that the case here.

To repeat : THOSE ARE NOT ERRORS! and they come from pkg(8) , not synth.

 

[marino]

So far so good, its going through the list one by one, but I do have one issue: if I roll my mouse wheel it causes a graceful shutdown. Arrgggg did it a couple times already. Other than that so far so good.

Only the escape key causes a shutdown, so unless your mouse wheel is causes escape sequences to go to the screen, I doubt that's it. Another way to think about it: Surely you aren't the only person moving a mouse? Nobody else is reporting this.

For my own education would be nice on your webpage to have a thorough explanation of what each build status means, maybe you can put that in the man page.

Those are ports build targets. There's pretty self explanatory IMO, unless you are talking about "idle" and "shutdown", which I think is also fairly obvious. In any case, it's just extra information to give insight into the build. Full understanding of exactly target the port is using isn't required. I think explanating port targets is out of scope for what I wanted to achieve with either the tutorial or the man page.

Also would be nice if the Built counter was split into two: Built-C (for ports/packages compiled locally on the machine), and Built-F (for packages fetched and extracted). Something for you to consider if you find yourself bored.

Hmm? Example: 10 packages should be built, and 6 are available. remotely. It should pull in those 6 and show "4" as the number to build. Is that not what is happening? For example, it's showing 10 left to build and completing after 4? If so, that's not intended. The "total" number is intended to be build-only, so that split you suggest doesn't make sense in that case.

Its too early to tell but overall progress seems to be considerably faster than pure postmaster compile for all. I'm estimating 10 hours at this point in time.

No, it's not too early. portmaster builds serially, one at a time. Synth can build 4, 6, even 10 packages simulateously (we've got it set 10 builders x 5 jobs each). There's no comparison here. Even using one or two builders for fetching while another 2 builders builder is a huge win over portmaster.

 

[marino]

marino@ ,
My standard procedure for updating my production servers, has been to use traditional jails on the development box I just updated with ports-mgmt/synth. Given my experience, aside from a couple of (not unexpected) huckups with ports-mgmt/synth was so good. I got to thinking how I might use synth as an update strategy for updating my servers. A couple of thoughts come to mind. On one hand, I could simply point synth to a different set of Ports, PortsConf, and Packages folders, using copies of those I already use for them within the jails I currently use to update them. But given the servers are on different different versions, I'm thinking -- especially where Linux related ports are concerned, that I might run into issues. So I was thinking it might be a better idea, if I ran synth within a jail itself. Seems fiesable, but thought I'd like to get your thoughts on the matter. Are either of my ideas better than the other, or niether, or...?

It's technically impossible to run synth in a jail because jails won't allow tmpfs mounts, so it's not an option.

Even if it was, I wouldn't recommend it. All your servers that share a common ABI (e.g. FreeBSD 10 amd64) can use the same repository. You only need to build it once and you can share it among many. If the release number or architecture is different, you'll need a unique repository for those.

A repository is just a menu. You can have a 5000-package local repository and only have 20 packages installed on the system that uses it. So having linux packages in a repository where you don't need them doesn't hurt anything.

 

[marino]

PacketMan

Only the escape key causes a shutdown, so unless your mouse wheel is causes escape sequences to go to the screen, I doubt that's it. Another way to think about it: Surely you aren't the only person moving a mouse? Nobody else is reporting this.

That being said, the choice of using ESCAPE key has already bitten, maybe something to do with ANSI codes for terminals. That issue was bandaged but maybe we should change it to capital "Q" because this isn't the first report of inadvertent shutdown. I chose it because it's an intuitive thing but it seems to have some undesired consequences.

 

[Chris_H]

to repeat : THOSE ARE NOT ERRORS! and they come from pkg(8), not synth.

Indeed. That was my point exactly. I apparently didn't articulate that very well. But I did also believe that pkg/synth felt they needed access to them. But my point was that the so-called "errors" were informative only. Oh well.

 

[Chris_H]

It's technically impossible to run synth in a jail because jails won't allow tmpfs mounts, so it's not an option.

Even if it was, I wouldn't recommend it. All your servers that share a common ABI (e.g. FreeBSD 10 amd64) can use the same repository. You only need to build it once and you can share it among many. If the release number or architecture is different, you'll need a unique repository for those.

A repository is just a menu. You can have a 5000-package local repository and only have 20 packages installed on the system that uses it. So having linux packages in a repository where you don't need them doesn't hurt anything.

Well, that was pretty much my assessment; save the inability to run within a jail.
As to the ABI differential; My development box is tracking -CURRENT, while the servers are currently on 9.3-STABLE. I'm aware of the repo hierarchy, and it's ability to harbor i386/amd64 branches, to an extent. But in the end I guess it's just not feasable to use synth for this. As to linux; My only concern in that regard was versioning (ABI) from say f10 9(.3) v. c6 (11) being an issue.

Thanks for the helpful reply, John.

--Chris

 

[marino]

But in the end I guess it's just not feasable to use synth for this. As to linux; My only concern in that regard was versioning (ABI) from say f10 9(.3) v. c6 (11) being an issue.

Why not?
Build a 9.3 world on current and install it with DESTDIR. Create a new profile that defines the system root at that destdir. voila, you have 11 building packages for earlier systems. The only gotcha are ports that run conftests that pass with 11 kernel but fail with 9.3 kernel. Then the packages produced would be nonfunctional on 9.3

 

[PacketMan]

Only the escape key causes a shutdown, so unless your mouse wheel is causes escape sequences to go to the screen, I doubt that's it. Another way to think about it: Surely you aren't the only person moving a mouse? Nobody else is reporting this.

Packetman@
That being said, the choice of using ESCAPE key has already bitten, maybe something to do with ANSI codes for terminals. That issue was bandaged but maybe we should change it to capital "Q" because this isn't the first report of inadvertent shutdown. I chose it because it's an intuitive thing but it seems to have some undesired consequences.

I'll do some testing but I'm pretty sure its happened twice: if I put my mouse arrow over the xterm session running synth and roll the wheel, synth pretty much instantly sticks that yellowish line in there "graceful shutdown in progress....".

 

[marino]

somehow you mouse is emitting escape key code. I think there's enough weird stuff happening to switch this to a capital "Q" which requires a two-key combination and will be hard to do accidently (unless some ansi code happens to trigger it)

 

[PacketMan]

Hmm? Example: 10 packages should be built, and 6 are available. remotely. It should pull in those 6 and show "4" as the number to build. Is that not what is happening? For example, it's showing 10 left to build and completing after 4? If so, that's not intended. The "total" number is intended to be build-only, so that split you suggest doesn't make sense in that case.

When I started I had a number like 656 in the Total field. As each one is 'built', whether fetched prebuilt packages, or source code is downloaded an compiled on machine into a package, the Left counter decrements by 1, the Built counter increments by 1. Coming back to my screen later, say after a nights sleep, I have no way of telling how many actually prebuilt packages were downloaded and used versus how many were source code compile on my machine. See what I mean? I'm just trying to get a sense of what percentage of ports used prebuilt packages instead of local compile.