Synth: Introducing new custom package repository builder for FreeBSD and DragonFly

[marino]

gkontos, thanks!
Don't worry, when 1.00 is released, I'll start a new thread and hopefully it won't get muddied up.
I was hoping 0.99_5 was the last release before 1.00, but it looks like there will be one more. (As a bonus, a new dev feature that I was saving for the next release could make it into 1.00)

 

[marino]

ports-mgmt/synth: bug fixes and feature for port developers

I had hoped that 0.99_5 would be bug-free and the basis for the first
release (1.00), but couple were found.  If use of 0.99_6 reveals no
further issues after a week or so, I'll re-release it as v1.00.

bugs fixed:
  * if the origin started with a directory separator, an exception would
    occur.  Now it properly labels it as an invalid origin.
  * The "extract" stage was labelled as "checksum".  Internally everything
    was fine, but on the display, the order was checksum, extract-depends,
    checksum instead of checksum, extract-depends, extract.
  * During one phase (build), the DEVELOPER flag was set unconditionally.
    This was a regression as it wasn't always the case.  This code was
    tweaked several times since 0.99_5 and now the DEVELOPER setting has
    been moved the builder's make.conf to ensure it's consistently
    present or absent (as needed).
  * It turns out that the ports tree scan is affected by the DEVELOPER
    flag.  It turns out the setting can affect the dependencies list so
    it needs to be set (or absent) appropriately to match how it will be
    on the builders.  The make.conf solution above solves this too.
  * There was "NO_BACKUP" set in the builders make.conf.  This line is
    for the DragonFly src builder and it's presence caused no harm, but
    it's been removed now.
  * Make ports makefile respect CFLAGS

new feature:
  * Provide ability to break into a build at a specific point and
    interact with it.
      -  Only available on "test" command
      -  Only active when one (1) port origin is given to "test" command
      -  Only active when ENTERAFTER is defined in environment as:
           > extract
           > patch
           > configure
           > build
           > stage
           > install
           > deinstall
      -  All dependencies are built first with typical display and
         DEVELOPER=1 set.  Afterwards, Synth converts to text mode and
         builds the specified port up to and including the phase specified
         by ENTERAFTER.  Then it launches a tcsh shell and gives control
         to the user at the builder's root directory
      -  The user ends the interactive session with the shell cmd "exit"
      -  Synth will clean up and exit (it will not try to continue the
         build due to possible corruption from the users)
   * This is a port developers tool.  The average user does not need it.
   * The average user might use "test" command to generate a log to submit
     as a FreeBSD Bugzilla PR attachment.

 

[basbebe]

Thank you so much marino@ for this great new tool!

I have two questions that I hope someone can help me with or point me in the right direction:

To use ccache and tmpfs, do I need to do anything besides installing ccache? Do I need to set specific settings in ccache or establish specific mounts for tmpfs?

If I build ports from a list to make them available to other systems (jails) via null mounts: How to make sure all dependencies for e.g. mariadb are considered correctly in the target system? Would I just add something in the LiveSystem-make.conf like WANT_MYSQL_VER= 100m and add all the packages from the target system that could need mysql to the list?

 

[marino]

Unless your machine is critically low on memory, you should already have tmpfs as the default (check with synth command). If synth defaults to tmpfs=off then you probably don't have much RAM so building with tmpfs may not be a good idea.

for ccache, you install it separately and configure it (per man page or type ccache for a list of commands). When it's configured, type ccache -s and use the value for cache directory (which you may have configured already) and put that directory in option [H] of synth configure. That's it, synth will use ccache after that.

If I build ports from a list to make them available to other systems (jails) via null mounts: How to make sure all dependencies for e.g. mariadb are considered correctly in the target system? Would I just add something in the LiveSystem-make.conf like WANT_MYSQL_VER= 100m and add all the packages from the target system that could need mysql to the list?

No, you only need one port origin (databases/mariadb100-server) on your list. Synth will build the dependencies and put everything in the local repository when you decide to build it. You only need to specify the leaf ports.

 

[basbebe]

for ccache, you install it separately and configure it (per man page or type ccache for a list of commands). When it's configured, type ccache -s and use the value for cache directory (which you may have configured already) and put that directory in option [H] of synth configure. That's it, synth will use ccache after that.

So I won't have to add anything to make.conf or LiveSystem-make.conf?

No, you only need one port origin (databases/mariadb100-server) on your list. Synth will build the dependencies and put everything in the local repository when you decide to build it. You only need to specify the leaf ports.

Will this also work if the host system doesn't have mariadb100 installed but it's only in the list? I know that it won't be linked correctly for the host system but will it still work for the targets?
And (hypothetically, just out of curiousity) what would happen if I had mysql56 and mariadb100 both on my list?

 

[marino]

So I won't have to add anything to make.conf or LiveSystem-make.conf?

no, you don't add anything to those files.

Will this also work if the host system doesn't have mariadb100 installed but it's only in the list? I know that it won't be linked correctly for the host system but will it still work for the targets?
And (hypothetically, just out of curiousity) what would happen if I had mysql56 and mariadb100 both on my list?

You can add whatever you want, and yes it always builds the dependencies.
A repository is a collection of packages. The host system (or any system) that uses the respository only installs a subset of the packages.

Extreme example: I can tell synth to build every port in the tree, and afterwards when I run the synth upgrade-system command, only a few packages are replaced. There's no relationship about what is installed on system and what's in the local repository. All the "prepare-system" and "system-upgrade" commands really do is give a port build list to synth so it can update / add those to the local repository.

more clear?

 

[basbebe]

You can add whatever you want, and yes it always builds the dependencies.
A repository is a collection of packages. The host system (or any system) that uses the respository only installs a subset of the packages.

Extreme example: I can tell synth to build every port in the tree, and afterwards when I run the synth upgrade-system command, only a few packages are replaced. There's no relationship about what is installed on system and what's in the local repository. All the "prepare-system" and "system-upgrade" commands really do is give a port build list to synth so it can update / add those to the local repository.

more clear?

Thanks, yes.
I'm still not entirely sure as to how I would replace mysql56-client with mariadb100-client.
If I had a list with e.g. dovecot2, postfix, php56-mysql and mariadb100-client, would all these packages built with mariadb100-dependencies?

 

[marino]

Unless you specify otherwise, they build with the default ports options.
There are two ways to customize options:
1) go to port and run make config and save the results
2) put in global options in the LiveSystem-make.conf file that you create.
You won't be able to support two different mariadb defaults in the same repository. All the systems that share a repository have the same package configurations (should be obvious).

 

[marino]

by the way, dovecot has no dependency on mariadb. php56-mysql probably doesn't either. So the answer is "no", just putting them in the same portlist does not influence dependencies in any way.

 

[basbebe]

by the way, dovecot has no dependency on mariadb. php56-mysql probably doesn't either. So the answer is "no", just putting them in the same portlist does not influence dependencies in any way.

Thanks for the clarification!

I put WANT_MYSQL_VER= 100m into LiveSystem-make.conf and now synth status /usr/local/etc/synth/jail_ports.txt shows:

These are the ports that would be built:
  => mail/dovecot2
  => databases/p5-DBD-mysql
  => databases/py-MySQLdb
  => mail/dovecot2-antispam-plugin
  => mail/dovecot2-pigeonhole
  => mail/opendmarc
  => mail/postfix-current

Thanks!

Looking forward to a great synth github wiki

 

[fernandel]

I have one ignored port, sysutils/fusefs-expat:

sysutils/fusefs-exfat: License Microsoft-exFAT needs confirmation, but BATCH is defined

Should I modify Makefile, please?
Thank you.

 

[hukadan]

Try adding the following line to make.conf.

DISABLE_LICENSES=yes

You can have a look to the /usr/ports/Mk/bsd.licenses.mk file for more details (and options).

 

[marino]

yes, I should probably add DISABLE_LICENSES to the main make.conf as well for release 1.00

 

[marino]

although maybe I shouldn't and just let people add "DISABLE_LICENSES" as needed. The current situation might be the best.

 

[marino]

yes, there's a LICENSES_ACCEPTED and LICENSES_GROUPS_ACCEPTED that would be appropriate to set.

 

[Uniballer]

I built ports-mgmt/synth and used it to build a set of ports I use on certain headless systems (amd64). I was very impressed at how smoothly it went. Nice work.

I discovered that I can clean up the mounts and directories from an interrupted build by running synth status. I didn't see that in the man page, did I just miss it?

Is there a way to tell synth to get rid of everything and start over? I mean, get rid of all of its live packages, logs, etc. I know I can delete them from /var/synth/live_packages/All and /var/log/synth, but is that really everything?

 

[marino]

any major command (read: not help or version, but everything else) will clean up dirty mounts if detected. Do you know about graceful exits? (Control-Q). Just breaking the build with control-C is not recommended. I don't remember if auto-cleanup is on the man page or not.

rm -rf /var/synth/live_packages/* should do it. Do you really care about the logs? They'll get overwritten. You could also remove /usr/local/etc/synth/* but that will blow away your configuration, so you probably don't want to do that.

 

[marino]

Uniballer, yes it is in the man page. See "NOTES" section, "Graceful Exit". There's a comment about cleaning up dirty mounts if the build was hard-interrupted.

 

[PacketMan]

You can add whatever you want, and yes it always builds the dependencies.
A repository is a collection of packages. The host system (or any system) that uses the respository only installs a subset of the packages.

Extreme example: I can tell synth to build every port in the tree, and afterwards when I run the synth upgrade-system command, only a few packages are replaced. There's no relationship about what is installed on system and what's in the local repository. All the "prepare-system" and "system-upgrade" commands really do is give a port build list to synth so it can update / add those to the local repository.

Sorry if this seems redundant: So can I

• have my fastest machine build all the packages using the synth just-build command?
• then copy required packages (or all of em) to the other machines repositories? (with a portsnap fetch update on each one too)
  
• then issue synth upgrade-system and it will use those freshly built packages instead of building locally from scratch, to upgrade only what is installed?
• in the event one machine has different make options configured, it will not use that package and create its own by building from scratch?

And suppose I want my 'master machine' to update (rebuild) all the packages using the new updates? Do I do a portsnap fetch update followed by a synth rebuild-repository then copy the packages over to the other machines, and then issue synth upgrade-system?

Its still not clear to me when I would use synth rebuild-repository versus synth prepare-system.

 

[marino]

have my fastest machine build all the packages using the synth just-build command?

Not normally. Normally you would only issue this command for ports that haven't been added to the system yet. Once they are on the host system (in pkg database) synth will rebuild *if necessary* with prepare-system or upgrade-system commands

then copy required packages (or all of em) to the other machines repositories (with a portsnap fetch update on each one too)

no. Why would you? Just tell synth the URL (file:///, http:///, etc)
it's possible I don't understand what you mean but it sounds very very wrong.


let me stand back and start again. It soulds like you want a common repo with multiple consumers. So in that case, you want a fixed list of port origins. That list represents what is required to be in the target repository.
You could script it so synth does a "just-build" with that list and it will rebuild incrementally.
Then the script would follow with a "rebuild-repository" command.

"system-prepare" and "upgrade-system" are commands only for the host system. In this scenario you wouldn't use them. You would just manually issue the pkg upgrade -a command on each repository consumer.

Did I get us both closer?

 

[PacketMan]

Did I get us both closer?

Not sure.

Compiling ports from scratch drives me bonkers. It is very time consuming, especially on my older machines, one port alone can take hours. So I was thinking I would have a fast server do all the building/compiling work. It would contain all the packages for whats installed on all the other machines (and including itself). Then when ports tree updates are available, I want to do any and all rebuilding on the fast server, then push the packages to the other machines, and then tell them to simply 'upgrade' to the new packages, with no local build/compiling required, unless of course a package can't be reused because make options are different.

I don't want the machines to share a single common repository. I want each to have its own, and I want to manually copy packages from the master fast server to the slower machines.

See what I mean?

 

[marino]

PacketMan
I was with you until you said "I want them all to have their own repo".
The normal scenario is have your fast machine build one repo and all the other machines would connect to it somehow.
You can duplicate it X times, but why would you? what's the benefit? you have to rsync everything... for a gain I don't see (unless the machines aren't connected by the same network, but then how do you copy gigabytes of data ...?)

 

[PacketMan]

PacketMan
The normal scenario is have your fast machine build one repo and all the other machines would connect to it somehow.
You can duplicate it X times, but why would you? what's the benefit? you have to rsync everything... for a gain I don't see (unless the machines aren't connected by the same network, but then how do you copy gigabytes of data ...?)

Bingo! My machines are not on the same LAN. I intend on using Bittorrent SYNC to do this. I can do my building on one machine one day, and then launch the upgrades on other machines another day. And in the event my main builder dies, I can use my next fastest machine to take over, or replace it with a brand new machine, and repeat the whole thing, and let Bittorent Sync to its magic!

So is there anything bad with what I want to do? I don't see it.

 

[marino]

it's inefficient, but it will work. The ideal is to have one common repository (and when the repo is rebuilt, the new packages are immediately available everywhere)

 

[PacketMan]

it's inefficient, but it will work. The ideal is to have one common repository (and when the repo is rebuilt, the new packages are immediately available everywhere)

Remember this phrase my freind, I learned it a long time ago and it has served me very well. "The best technical solution is not necessarily the best business solution."

I will test and post later on the forum, likely in a new discussion topic. Thanks again Marino for an awesome product. You have give me new hope and joy.