Speed up the building of ports using prebuilt packages?

  • Thread starter Thread starter Deleted member 66267
  • Start date Start date
D

Deleted member 66267

Guest
Most of the time spent is not of building the actual port itself, but the dependencies of it.

Since I only modify the build options of just this port, could I just grab prebuilt packages for the dependencies to speed up the build?
 
chrbr said:
portmaster(8) has now an option to use packages for build dependencies. There are a few other package related options, too. Please check the man page and the options starting with --packages.
Click to expand...
Thank you. But I like SirDice's answer more because I hear the tool you mentioned was broken and deprecated for a long time in favor of poudriere(8).
 
Dear also-ran,
if you have a machine which can run poudriere(8) this is the best option. Some ports take a long time to build. Therefore the demands on the build machine is steadily increasing. I own old stuff ;-).
But even when you have a weak or medium powerful machine it is worth to learn and run poudriere(8) because it works well (beside the build time issues) and it is the offical and most secure way to build packages.
 
For package building you might also want to take a look at ports-mgmt/synth. It uses the same principles (building packages in a "clean" environment) but works a little different. Its the package build system that's used by DragonFlyBSD. Poudriere is the de facto standard on FreeBSD because it's used on the FreeBSD build clusters. Both tools will allow you to create your own package repositories. The benefit is that you can use the flexibility of the ports, by enabling or disabling options and setting your own defaults (default MySQL version, PHP, Perl, Python, Ruby, etc), while keeping the ease of management from packages (install, upgrade, etc).
 
Is it possible to pre-build MySQL 8.0 dependencies (MySQL 5.7 already installed) and mark them as automatically installed and then remove MySQL 5.7 and build MySQL 8.0 to avoid the most downtime?
 
CyberCr33p said:
Is it possible to pre-build MySQL 8.0 dependencies (MySQL 5.7 already installed) and mark them as automatically installed and then remove MySQL 5.7 and build MySQL 8.0 to avoid the most downtime?
Click to expand...
It depends on what you mean by 'avoiding downtime'. The process would be much simpler if you just build MySQL 8.0 from ports, and let the dependency resolution take care of the rest. On your end of things, does the build process make the rest of the computer go unacceptably slow? It takes about 5 minutes on a Ryzen 5 1400 3.4 GHz to compile MySQL and dependencies. If you want to build MySQL 8.0 (but not install it), all you need to do in /usr/ports/databases/mysql80 is # make, with no other arguments.
 
If you go with the package route by building your own packages then you don't typically build those packages on the production server anyway. So the building itself won't cause any downtime. The database will only need to be taken offline when you actually install the resulting package on the machine.
 
Even then, the package database is file-backed. Off the top of my head, I think it's BDB, not something used in the same manner as MySQL. The system only needs a few seconds to actually install the packages. Even big ports take less than a minute to install.
 
astyle said:
Even then, the package database is file-backed.
Click to expand...
Has nothing to do with it. When you build databases/mysql80-server, the dependency databases/mysql80-client has to be built and installed first. If you do this on the server that has the actual MySQL 5.7 running installing databases/mysql80-client would remove databases/mysql56-client (conflicting port) and as a consequence it will remove databases/mysql57-server too. You're then in "limbo" mode where the service is still running but the actual executables have been removed from the system. Not a situation you want to be in. So you shutdown the database before starting the build and would then be offline until you can get MySQL 8.0 server to build and install.

astyle said:
Off the top of my head, I think it's BDB
Click to expand...
pkg(8) uses an SQLite database.

astyle said:
The system only needs a few seconds to actually install the packages.
Click to expand...
Yes, exactly. Create a package from the port on a different system (using poudriere, synth or by hand), transfer the packages and install those on a test system first (to verify the package actually works). You can also test the upgrade from 5.7 to 8.0 here, that way you know in advance what kind of issues you might run into. Once you're satisfied it works you install those packages on the production server. That only takes a couple of minutes max.
 
in theory you can pkg delete 5.x client and server install 8.x client and server and restart
server continues to run even if the binary is gone
if some error message files are not mmaped and the server needs them in the interim period that may be a problem
 
SirDice said:
You're then in "limbo" mode where the service is still running but the actual executables have been removed from the system. Not a situation you want to be in.
Click to expand...
This idea always feels really nasty but pretty much happens every day to a large proportion of desktop users.

Lets say someone is running a Gnome or KDE desktop. They run a pkg upgrade and it pretty much just strips out the old executables and resources and just shoves in new potentially different ones. The old executables are still running and loaded in memory and accessing resources that aren't intended for them. There could even be weird logic that replaces files with original versions if it doesn't "understand" them so half of the new files could potentially be replaced with half old ones again.

Package managers are superior to the ad-hoc Windows way but still seem to benefit with a good ol' restart after ;)
 
kpedersen said:
This idea always feels really nasty but pretty much happens every day to a large proportion of desktop users.
Click to expand...
I do this quite frequently actually but not with 'important' services like databases. With those I just feel safer shutting them down first, then do the update. Any other service is usually easily fixed if you do run into issues but with databases I'm always a bit more careful.
 
also having the server port installing the client is a bug for mysql at least
Code: 
[19:06:41] [host!root]~#pkg delete -f mysql57-client
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 1 packages (of 0 packages in the universe):

Installed packages to be REMOVED:
    mysql57-client: 5.7.36

Number of packages to be removed: 1

The operation will free 49 MiB.

Proceed with deinstalling packages? [y/N]: y
[1/1] Deinstalling mysql57-client-5.7.36...
[1/1] Deleting files for mysql57-client-5.7.36: 100%
[19:06:44] [host!root]~#/usr/local/etc/rc.d/mysql-server stop
Stopping mysql.
Waiting for PIDS: 50129, 50129.
[19:06:58] [host!root]~#mysqladmin -u root -p status^C
[19:07:06] [host!root]~#/usr/local/etc/rc.d/mysql-server start
Starting mysql.
[19:07:11] [host!root]~#pkg install -f mysql57-client
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
    mysql57-client: 5.7.36

Number of packages to be installed: 1

The process will require 49 MiB more space.

Proceed with this action? [y/N]: y
[1/1] Installing mysql57-client-5.7.36...
[1/1] Extracting mysql57-client-5.7.36: 100%
=====
Message from mysql57-client-5.7.36:

--
This is the mysql CLIENT without the server.
for complete server and client, please install databases/mysql57-server
[19:07:17] [host!root]~#mysqladmin -u root -p status
Enter password:
Uptime: 15  Threads: 1  Questions: 2  Slow queries: 0  Opens: 108  Flush tables: 1  Open tables: 101  Queries per second avg: 0.133
[19:07:25] [host!root]~#
server can definitely run without the client
 
The client has been a run dependency for the server for as long as I can remember. It's possible it's not actually a run requirement but the port/package certainly has a build and run dependency on it.

Code: 
root@molly:/usr/ports/databases/mysql80-server # make run-depends-list
/usr/ports/ftp/curl
/usr/ports/devel/libevent
/usr/ports/devel/icu
/usr/ports/archivers/liblz4
/usr/ports/archivers/zstd
/usr/ports/devel/protobuf
/usr/ports/devel/libunwind
/usr/ports/devel/libedit
/usr/ports/databases/mysql80-client
/usr/ports/textproc/groff
/usr/ports/lang/perl5.32
root@molly:/usr/ports/databases/mysql80-server # make build-depends-list
/usr/ports/ports-mgmt/pkg
/usr/ports/archivers/liblz4
/usr/ports/devel/libunwind
/usr/ports/devel/bison
/usr/ports/devel/cmake
/usr/ports/devel/pkgconf
/usr/ports/ftp/curl
/usr/ports/devel/libevent
/usr/ports/devel/icu
/usr/ports/archivers/zstd
/usr/ports/devel/protobuf
/usr/ports/devel/libedit
/usr/ports/databases/mysql80-client
 
USES=mysql:80 implies client
also port forbids version mismatch between client and server which is a lesser problem
 
SirDice said:
The client has been a run dependency for the server for as long as I can remember. It's possible it's not actually a run requirement but the port/package certainly has a build and run dependency on it.

Code: 
root@molly:/usr/ports/databases/mysql80-server # make run-depends-list
/usr/ports/ftp/curl
/usr/ports/devel/libevent
/usr/ports/devel/icu
/usr/ports/archivers/liblz4
/usr/ports/archivers/zstd
/usr/ports/devel/protobuf
/usr/ports/devel/libunwind
/usr/ports/devel/libedit
/usr/ports/databases/mysql80-client
/usr/ports/textproc/groff
/usr/ports/lang/perl5.32
root@molly:/usr/ports/databases/mysql80-server # make build-depends-list
/usr/ports/ports-mgmt/pkg
/usr/ports/archivers/liblz4
/usr/ports/devel/libunwind
/usr/ports/devel/bison
/usr/ports/devel/cmake
/usr/ports/devel/pkgconf
/usr/ports/ftp/curl
/usr/ports/devel/libevent
/usr/ports/devel/icu
/usr/ports/archivers/zstd
/usr/ports/devel/protobuf
/usr/ports/devel/libedit
/usr/ports/databases/mysql80-client
Click to expand...
A server needs a client installed at least locally, but not the other way around. Gotta be careful about that when specifying the deps - direction of the dependency really matters.
astyle said:
If you want to build MySQL 8.0 (but not install it)
Click to expand...
See the rest of my post. This is the nice thing about forums - you can easily go back and see what was said on the topic earlier - it just requires a bit of attention. ?
 
I don't use poudriere.

For minor MySQL upgrades I use portmaster while MySQL running and then I restart to reload the new binary. But for major upgrades we do a full mysqldump, a rsync of /var/db/mysql while MySQL is running, then shudown MySQL, make a new rsync which doesn't take much time, then a "pkg del" for old version, then I do the upgrade to the new major version (if I use portmaster then it needs ~20 minutes to complete because it needs to install the "build" dependencies such as python), I replace the my.cnf with the one that is compatible with new version, and then I start MySQL. MySQL 8.0 doesn't need mysql_upgrade any more as it takes care of it during the MySQL start.

I create mysql80-server-8.0.27.pkg and mysql80-client-8.0.27.pkg as the server I originally build it and all other servers have the same software & versions installed (and all MySQL running dependecies already installed) so now instead of portupgrade I use "pkg add mysql80-server-8.0.27.pkg" (mysql80-client-8.0.27.pkg automatically installed as both files are in the same directory).

So instead of 20-25 minutes downtime I now have 1-2 minutes downtime per server.
 
CyberCr33p said:
I don't use poudriere.
Click to expand...
CyberCr33p said:
and all other servers have the same software & versions installed
Click to expand...
Sounds like you really need to set up your own repository. Honestly, look into it. It's going to make updates and upgrades like this so much easier. Because you can have everything you need already built, ready to be installed from a central, local, repository. And now you've changed to MySQL 8.0 you really want to set DEFAULT_VERSIONS+= mysql=8.0 and have everything else depend on the right version of the client too.
 
You must log in or register to reply here.
Back
Top