Ports with PKGNG installed dependencies, problems

[honeybaker]

I'm very new to FreeBSD and I realize now that I've made some bad decisions as far as installing packages goes.

I'm using 9.2-RELEASE on a headless server.

My /usr/local/etc/pkg/repos/FreeBSD.conf looks like this:

FreeBSD: {
  url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest",
  mirror_type: "srv",
  enabled: yes
}

Example of something I've done:

1. I installed lang/php5-extensions using ports.
2. I installed certain dependencies for it using pkg.

There are a few other packages that I used the same process with and it didn't cause any problems when I did it at the time but later I realized that I did something wrong when I wanted to upgrade packages. When I execute pkg upgrade now, it wants to reinstall a lot of packages (93 package upgrades in total at this time), ones I also installed using ports because "direct dependency changed". I haven't accepted any upgrade or reinstall yet because I expect it will cause issues since I expect it to install the binaries from the pkg repository which won't have all the modules and extensions I need in some packages.

I hope there's an easy way out of this predicament :\

Please keep in mind that I'm very new to all of FreeBSD.

Thank you.

 

[SirDice]

Stick to either packages or ports. Don't try to mix them until you know what you are doing.

 

[kpa]

This situation may change when ports-mgmt/pkg version 1.3 gets released, the developers have hinted that by that time it should be possible to mix the official packages and self built ports/packages.

 

[honeybaker]

Stick to either packages or ports. Don't try to mix them until you know what you are doing.

I'll go with ports but what should I do now that I'm in this situation?

 

[junovitch@]

If you know that the particular mix of ports and packages won't cause issues, pkg lock allows locking packages that you don't want pkg to upgrade.

 

[honeybaker]

If you know that the particular mix of ports and packages won't cause issues, pkg lock allows locking packages that you don't want pkg to upgrade.

I don't know if any of the mixes will or won't cause issues. Is there a way I can find out which packages were installed using ports? It's not that I have a terrible memory but I would like to be sure. I don't really want to check each of the 238 packages I have installed one by one.

 

[junovitch@]

Anything you installed from ports and set options for will be in /var/db/ports with an options file for what you set. You can use that to jog your memory on what was installed through ports. If you remember what you customized, you can try using pkg lock <package>. According to the man page, (see with pkg help lock) it will lock that package and any upgrades or changes that imply changing it. Just be warned that you could end up somewhere unpredictable if you do that and start mixing ports and packages. If you have a bunch of customizations, it would be safest to use what you learn from your options files in /var/db/ports to build your own repository with everything you need and just re-install everything at once via pkg upgrade -fy.

 

[wblock@]

Unless you have multiple machines that will use the same packages, why not just build from ports? If you set up a custom repository, you'll end up building those packages from ports anyway.

 

[honeybaker]

Unless you have multiple machines that will use the same packages, why not just build from ports? If you set up a custom repo, you'll end up building those packages from ports anyway.

How would a newbie go about doing that, setting up a custom repository with all the packages I will need or might need in the future?

I'm looking at https://fossil.etoilebsd.net/poudriere/ ... index.wiki and the how-tos seem to be targeted at more advanced setups with multiple machines that have lots of RAM, I don't have the confidence to try and make them work for my situation. I only have access to the one machine which has 2 gigabytes of RAM.

What do I do with the 238 packages I currently have installed (most using pkg, with the repository I mentioned above)? I want to end up with the same packages but from ports, some with custom options which I see are stored in /var/db/ports. Do I run the risk of overwriting configurations for packages like zsh, znc, php, and nginx if I end up having to remove all of my 238 packages and installing them afterwards?

Thank you.

 

[wblock@]

I say again, why not just build from ports? 238 ports is not a huge number. Unless you have a reason to build packages, like sharing them on multiple machines, setting up your own repository is just needless overhead. portmaster(8) shows a procedure for rebuilding all ports without losing the existing configuration.

 

[honeybaker]

I say again, why not just build from ports? 238 ports is not a huge number. Unless you have a reason to build packages, like sharing them on multiple machines, setting up your own repository is just needless overhead. portmaster(8) shows a procedure for rebuilding all ports without losing the existing configuration.

Apologies, I read that wrong.

The procedure on that page is a little different from what I get with man portmaster on my install though:

Using portmaster to do a complete reinstallation of all your ports:

• 
  1. portmaster --list-origins > ~/installed-port-list
  2. Update your ports tree
  3. portmaster -ty --clean-distfiles
  4. portmaster --check-port-dbdir
  5. portmaster -Faf
  6. pkg_delete -a
  7. rm -rf /usr/local/lib/compat/pkg
  8. Back up any files in /usr/local you wish to save, such as configuration files in /usr/local/etc
  9. Manually check /usr/local and /var/db/pkg to make sure that they are really empty
  10. Re-install portmaster
  11. portmaster `cat ~/installed-port-list`
  
  You probably want to use the -D option for the installation and then run --clean-distfiles [-y] again when you are done. You might also want to consider using the --force-config option when installing the new ports.
  Alternatively you could use portmaster -a -f -D to do an ``in place'' update of your ports. If that process is interrupted for any reason you can use portmaster -a -f -D -R to avoid rebuilding ports already rebuilt on previous runs. However the first method (delete everything and reinstall) is preferred.

I will want to back up some stuff from /usr/local/etc. But should I execute pkg_delete -a though, since I've been using PKGNG? Should I be using --force-config, won't that make me have to set options for all of my 238 packages, would that be necessary? Will the options I've set in the past, that are saved in /var/db/ports be used when It's reinstalling? Will I have to stop my running processes while I do this procedure, e.g. my web server and IRC bouncer?

Thank you so very much.

 

[wblock@]

The procedure on that page is a little different from what I get with man portmaster on my install though

The online one is an older version, use the newer one from your system.

I will want to back up some stuff from /usr/local/etc.

I'd suggest backing all of it up. In fact, before a major change, back up the whole system. See Backup Options For FreeBSD.

But should I execute pkg_delete -a though, since I've been using pkgng?

No, you're right, use pkg delete -a.

Should I be using --force-config, won't that make me have to set options for all of my 238 packages, would that be necessary?
Will the options I've set in the past, that are saved in /var/db/ports be used when It's reinstalling?

Any ports you installed keep the same options as when you installed them. Packages used the default options anyway.

Will I have to stop my running processes while I do this procedure, e.g. my web server and IRC bouncer?

You don't have to stop them, but after deinstalling everything, the program files will be gone and there's a very good chance for problems until the new binaries have been built and installed. If downtime is a serious concern, you can use a VM or a second machine. Back up the original, restore on the VM, do the whole upgrade. If there are problems, note them and continue. This can be a dress rehearsal, or once completed, you can back up the VM and restore it to the original.

 

[honeybaker]

I created a ZFS snapshot of my entire system. I also backed up a few directories manually just to be sure, by copying them to my local machine.

Everything worked out, it took me four hours to compile everything. I couldn't contain my excitement when everything started back up and worked like I never even reinstalled every single package from a different source. How do I buy you a beer, @wblock@?

PS I'm sorry about the improper formatting, I hope this post is correct.

 

[rustyx]

I have a similar issue (of mixing pkg and ports) but a more general question.

I installed openjdk7 and tomcat7 via ports.

Now pkg upgrade wants to install openjdk6 and reinstall the same version of tomcat7.

Obviously I have no interest in installing openjdk6 (even if it's just to make pkg happy).

My question is - is it possible to tell PKG to use a certain version of a dependency or to somehow keep the dependency calculation rules in sync between ports and pkg? Because if I install tomcat7 from ports then it happily takes the existing openjdk7 (and is working perfectly fine with it of course).

 

[SirDice]

I installed openjdk7 and tomcat7 via ports.

Now "pkg upgrade" wants to install openjdk6 and reinstall the same version of tomcat7.

That's because it was built using the default options, and that means openjdk6.

My question is - is it possible to tell PKG to use a certain version of a dependency or to somehow keep the dependency calculation rules in sync between ports and pkg?

No, that's not possible. At least not yet. Dependencies of packages are set in stone, you cannot change them.

Because if I install tomcat7 from ports then it happily takes the existing openjdk7 (and is working perfectly fine with it of course).

Set up you own repository, then you can set whatever you want as default.

 

[kpa]

My question is - is it possible to tell PKG to use a certain version of a dependency or to somehow keep the dependency calculation rules in sync between ports and pkg?

No, that's not possible. At least not yet. Dependencies of packages are set in stone, you cannot change them.

Yes, not yet possible because such functionality was never even planned for the older pkg_* tools and the new PKG tools that we now have more or less replicate what the older tools did in terms of functionality but of course with much more solid and robust foundation. The upside of the new PKG tools is that they make it possible to develop them further and this exact functionality that allows picking and choosing dependencies at install time (often completely automatically) is planned for the next release 1.3 of ports-mgmt/pkg.

 

[rustyx]

Dependencies of packages are set in stone, you cannot change them.

Well that's unfortunate as reality is more complex than "this package depends on that one, point". The dependencies system of pkg needs to be extended to at least support version ranges, for example to be able to express that tomcat7 runs with openjdk6 as well as openjdk7. Even if it's built with java 6, it runs perfectly fine with java 7 without rebuilding.