Pickled Situation...Any advice is welcomed

[raism]

Hello all,
I made this account here, looking for any advice possible. Literally this is my very first post here.
My experience with FreeBSD is minuscule. Give or take 3 years, which is as long as I have been working in my current role. I have been a Linux user since 1997 and working as administrator since 2017. So that my brief introduction about me.

I have inherited a very complex infrastructure based on FreeBSD OS only and more than 200+ hosts bare metal and virtual hosts combined. We (which really is me) manage all this with poudriere and SaltStack. This is where the "pickled situation" comes in.

Obviously we have many many hosts in key roles running on FreeBSD 12.2. Obviously we have just enough servers with FreeBSD 13.2, fewer with 14.2 and just one on 15.1. Unfortunately, some hosts, must stay in the version of the OS that they have because of various demands on pkg versions. From Python, to C plus plus or from Mongodb to Mysql and Drupal with Nginx and PHP. I am maintaining all of those variations with Poudriere ports-tree, and by saying maintaining I mean, "I do not touch". If I upgrade my host who is my Salt-master it will break the communication to many many hosts with older salt version. Test hosts that I created with newer than FreeBSD 14.3 which have newer salt version, do not communicate with my Salt-master(the issue is the PKI key). On the other hand if I upgrade hosts with older version of MongoDB, or Python, it will break a dozen or more hosts with an in-house business critical applications.

I need to move forward, and every step I make it is a "GIANT-LOCK" that I have somehow to break, which will have a domino effect. What ever server we have in FreeBSD 14.2, I have put my sleeves up and through a very extensive hard work, I made it possible to happen.

So what would you to do to move forward? What would be your advise?
Please excuse my English, as it is not my native language and when I am in situations like this I might speak or write like here. I refuse to ask any AI agent, as still believe in HI, that's why I humble ask you if you are in a mood for trolling, or telling me to rtfm please do not.

Many thanks,
Raism

 

[SirDice]

Unfortunately, some hosts, must stay in the version of the OS that they have because of various demands on pkg versions.

The version of the OS has nothing to do with the versions of ports/packages. There is only one ports tree and all (supported) versions of FreeBSD use that exact same ports tree.

So what would you to do to move forward? What would be your advise?

FreeBSD 12.2, 13.2 and 14.2 are all EoL and not supported any more. I would get them to at least 14.4, or better 14.5 (is just around the corner). And do some testing with 15.1 in te mean time. The whole 14 major branch will be EoL in 2028, so prepare in advance.

 

[freezmi]

and by saying maintaining I mean, "I do not touch"

you cannot learn if you don't do mistakes.

I need to move forward, and every step I make it is a "GIANT-LOCK" that I have somehow to break, which will have a domino effect. What ever server we have in FreeBSD 14.2, I have put my sleeves up and through a very extensive hard work, I made it possible to happen.

So what would you to do to move forward? What would be your advise?

divide and conquer - slowly move the servers towards 14.x.
start with low hanging fruits - lower priority servers or ones that take the least effort to migrate and slowly you'll get braver

migrations should not happen on the server that has to be replaced. set up new hardware and work on it until it provides the same set of features as the server that it will replace without any noticeable regressions. switch them when 'ready'. worst case you can fall back onto the old server if you missed something.
after x amount of time reuse this old server as 'new hardware' to replace another one, and so on.

by the end you'll know 101 ways to break things - and how to avoid these mistakes.

best of luck, it sounds like a nice challenge!

 

[rbranco]

I suggest replicating the infrastructure in virtual machines as much as possible.