[Guide] Using Git to manage ports, source and documentation.

ShelLuser

Son of Beastie

Reaction score: 2,054
Messages: 3,770

Hi gang!

Disclaimer: I am honestly a little excited about recent developments so expect to find some (small) opinionated parts in this guide. Nothing excessive mind you, but I can sometimes get a little carried away and despite some believes I never really plan guides like this.

Editorial

In April 2021 the FreeBSD project has finished its full conversion from Subversion to Git, something which had been brewing for quite some time. As a result many people who are quite familiar with Subversion may have to re-learn a few things. And there will be some who don't see any advantages here, what's this big fuss all about? Trust me: there are many advantages, some are huge in my opinion (but please keep my disclaimer in mind, thanks!), but if that also applies to you is something I obviously can't say for sure.

And before I continue: there are also disadvantages, no question about it; I've seen it myself (and I'll share them too). See, just because these don't bother me doesn't mean I don't recognize the possibility that these can still bother others. We'll address them, no worries.

But honestly you guys.. I'm a little excited here. I just finished the conversion on my main server last night (I also stopped using portsnap in favor of git) and I like where this is going. Well, I hope that after reading my guide you may also find some advantages that can help you out. And of course I also hope the guide as a whole will be useful to you guys.

As always I'm going to try and cover the full deal, so if you come across a section which you already know about then you can easily skip it; I'll make sure to keep them (mostly) separated so they don't rely on each other.

What is Git? (and what should I do with it? 😉)

Git is a so called Version Control System ("VCS") which can help people to keep control over a project. A project can be anything; a single shell script, a group of shell scripts, the source code for a program (or many programs) and you can even maintain a kernel with it. Heck, Git doesn't even mind the inclusion of binary files (but now I'm getting a little ahead of myself).

Every time something changes in a project then these changes can be documented and stored using the VCS. As a result the VCS will keep track of the projects history, which gives you full control over every change. See, it doesn't only document these changes; it also records them meaning that you can easily go back in time and bring your project into a state it had before. So if it turns out that a certain change (or addition) was bad then it won't be a problem: just go back, remove the changes and optionally re-do any other additional changes and you managed to clean up your project without having to re-create or re-write dozens of changes.

So how does this work? Well, the VCS maintains a database in which it stores information about all the changes and/or additions to the project. By default Git uses the .git directory for this: you'll find it in the root directory of almost every project that is under version control. If not a directory then it'll be a .git file which contains a pointer to that directory. And as I mentioned earlier this project could be anything: from a single file to a whole collection of files and/or directories. We refer to a project that is under version control as a repository.

What makes Git stand out from the rest is its decentralized design. Or put differently: Git will always use a local database for the project. It doesn't matter if you started the project yourself or are merely copying one (in Git terms this is called "cloning"): you'll always end up with a fully functional project repository and all the (dis)advantages that come with it no matter what. See, one possible disadvantage is that this means you'll get the entire backlog of the project too and that is going to gobble up storage space. But a possible huge advantage could be the fact that you can easily share such a cloned project with others as well.

For example: my server is fully under "Git control" meaning that I use Git to maintain the Ports collection, the source tree and the FreeBSD documentation project. I also use a second (smaller) FreeBSD server as backup (backup MTA & DNS). So once I updated the Ports collection (or source tree!) on my main server I don't have to waste precious bandwidth to do the same on my other server. Naah, since both my servers are part of a virtual LAN I simply clone the repository directly from my main server. Easy!

So in my situation the (small) excess in storage space gets made up for with a (huge) reduction in bandwidth. Of course... if you only maintain one server all you're left with is that excess in storage space, I feel you... but trust me: there are stilll more advantages here.

Now that we roughly know what Git is all about let's take a look at how we can use this for FreeBSD systems administration.

The Ports collection

(brief) description/intro

(ey, I like to make my guides useful for newbies & veterans alike, just skip this part if you're a veteran)

The Ports collection is a collection of "blueprints" (as I like to call it) which can help you to install external software onto FreeBSD. Every Port contains a "blueprint" (a Makefile) which will tell the system how to obtain the software, how to prepare the software (usually this means compiling the source code) and finally how to install it (the system creates a package which is then installed using ports-mgmt/pkg). Keep in mind that the ports collection is most useful if you need (or want) to use the software with very specific customizations. For example: by default Git provides support for CVS, a GUI and even a web interface. If you don't need that functionality you could build Git using the Ports collection, de-select these options and then build & install Git. Now you'll have a Git version without that "bloat".

But if you don't need any of these customizations then you're likely much better off using FreeBSD's package manager. In other words: instead of using # make -d /usr/ports/devel/git install clean you'd use: # pkg install git. It'll also be much quicker! 😉

Keep in mind: mixing ports and binary packages is a very bad idea! So always use one method or the other, not both. If you wonder why then this guide might be a good read.

Installing the Ports collection (with Git)

As of April 2021 the FreeBSD project is using a dedicated server for their Git repositories: git.freebsd.org. And all we need to know here is that Git allows the usage of "accessible" protocols such as HTTP and HTTPS (and FTP(S), SSH, SMB, NFS, it even has its own GIT protocol! 😎). Another important detail is that it's custom to provide a repository using the .git extension, even though you're actually sharing a directory. You got to admit it does make things easier to recognize.

This is honestly all we need to know (provided that we're familiar with Git of course).

So, to install a new (fresh) ports collection you'd use: # git clone https://git.freebsd.org/ports.git /usr/ports. Done!

This will install the Ports collection (with its entire backlog) into /usr/ports. Once this is done you can use the Ports collection as you always have. Need to update it? Easy, just use: git pull within the /usr/ports directory to "pull" any optional updates into your local repository.

Changing from Subversion to Git

This can become a little tricky, especially if you also maintain subdirectories like distfiles and packages and would like to keep these. But don't worry, that's why this guide exists!

The easiest approach here is to move these two directories out of the way, fully delete the contents of /usr/ports and then grab the Ports collection as I mentioned above. If you're using ZFS and haven't set up separate datasets for these two then now might be a good idea to do so. Why? Well, for starters because this would allow you to unmount these datasets for now using # zfs unmount zroot/ports/packages and then continue with the above procedure.

But there's more to this story...

ZFS Ports collection storage management
Code:
peter@vps:/usr/ports $ zfs list -r zroot/ports
NAME                    USED  AVAIL  REFER  MOUNTPOINT
zroot/ports            6.68G   105G  1.05G  /usr/ports
zroot/ports/distfiles  2.73G   105G  2.73G  /usr/ports/distfiles
zroot/ports/packages   2.89G   105G  2.89G  /usr/ports/packages
As you probably know the bulk of /usr/ports is used by (ASCII) (Make)files (and patches). And text files are - generally speaking - very easy to compress. Since ZFS provides you with file compression out of the box, why not use it?
Code:
peter@vps:/usr/ports $ zfs get compression zroot/ports
NAME         PROPERTY     VALUE     SOURCE
zroot/ports  compression  on        local
All it takes is one command: # zfs set compression=on zroot/ports (keep in mind that you'd have to replace zroot/ports with your own dataset name). Also important to know: this won't magically compress every file which already exists, only those you copy onto this dataset from now on. Therefor it makes sense to use this option when you create the new dataset (or directly afterwards).

Now, more important details: distfiles (and optionally packages) contain archives. distfiles is used to store the archive(s) which were downloaded by the system in order to provide you with the actual software which the ports refers to. Remember: if you're building a port like, say, the Apache webserver then the system will start by downloading the source code from the Apache website, (temporarily) install that onto your system and then build the port. And in order to prevent the system from having to download the source code again if you need to re-build the port it keeps the downloaded archive in the distfiles directory.

packages on the other hand is often used by ports-mgmt/portmaster (and some others). In short this can be the ideal location to set up your own package repository which you can then provide to other servers in your network. Portmaster on the other hand uses this location to (optionally) store backup packages. When I upgrade a port on my system then Portmaster will keep the previous version in this directory, so should something go wrong I can always go back to the previous (working) setup no questions asked.

So the point here is that these two directories contain archives, and those are already compressed. It would be a waste of system resources if we'd let ZFS compress these files as well. And as you might know: ZFS properties propagate. In other words: properties set for a dataset will also apply to its children.

Therefor...
Code:
peter@vps:/usr/ports $ zfs get -r compression zroot/ports
NAME                   PROPERTY     VALUE     SOURCE
zroot/ports            compression  on        local
zroot/ports/distfiles  compression  off       local
zroot/ports/packages   compression  off       local
... you should make sure to turn compression off for any child datasets (provided that you set things up in the same way as I did of course).

But what if my /usr/ports directory isn't empty?!

So yeah, that's the ideal situation above but as we all know situations are usually far from ideal 😒. So there's good news and bad news for you...

The bad news is that Git will refuse to clone a repository into a directory which isn't empty:
Code:
peter@vps:/home/peter/temp $ mkdir -p myports/distfiles
peter@vps:/home/peter/temp $ git clone https://git.freebsd.org/ports.git myports/
fatal: destination path 'myports' already exists and is not an empty directory.
Now what?! Yah, to make matters worse (I'll spare you the time to study git-clone(1) (for now 😉)): you won't find any overrule or "force" options to use with the cloning process. But don't worry, there's a solution!

Important: When I refer to a 'ports' directory which isn't empty I'm only referring to the possible existence of additional subdirectories (as mentioned above) or (hidden) file entries as used by the UFS filesystem. I strongly suggest you do not try this approach in order to save you some download time (for example by trying to clone the Git ports repository into an existing Subversion repository). I'm not saying it won't work (I honestly don't know because I never bothered trying) but I do foresee a lot of problems if you'd try. Feel free to prove me wrong though 😉

Now the good news is that we can "trick" Git to actually bypass the cloning process:
Code:
peter@vps:/home/peter/temp $ cd myports/
peter@vps:/home/peter/temp/myports $ git init
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint:   git config --global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint:   git branch -m <name>
Initialized empty Git repository in /home/peter/temp/myports/.git/
peter@vps:/home/peter/temp/myports $ git remote add origin https://git.freebsd.org/ports.git
peter@vps:/home/peter/temp/myports $ git fetch origin
remote: Enumerating objects: 15730, done.
remote: Counting objects: 100% (15730/15730), done.
remote: Compressing objects: 100% (417/417), done.
Receiving objects:   4% (199650/4991230), 45.42 MiB | 10.09 MiB/s
[...]
* [new branch]                2020Q3     -> origin/2020Q3
* [new branch]                2020Q4     -> origin/2020Q4
* [new branch]                2021Q1     -> origin/2021Q1
* [new branch]                2021Q2     -> origin/2021Q2
* [new branch]                main       -> origin/main
* [new tag]                   10-eol          -> 10-eol
* [new tag]                   7-eol           -> 7-eol
* [new tag]                   8-eol           -> 8-eol
* [new tag]                   9-eol           -> 9-eol
* [new tag]                   pkg-install-eol -> pkg-install-eol
[...]
peter@vps:/home/peter/temp/myports $ git merge origin/main
peter@vps:/home/peter/temp/myports $ git branch --set-upstream-to=origin/main
Branch 'master' set up to track remote branch 'main' from 'origin'.

So to summarize:
  • Create a new empty repository: git init.
  • Add the remote repository and call it origin: git remote add origin https://git.freebsd.org/ports.git.
  • Fetch all available information about the remote repository: git fetch origin.
  • Merge the remote repository main 'into' our local branch: git merge origin/main.
  • Make our local (master) branch track origin/main: git branch --set-upstream-to=origin/main.
Now what is happening here? Well, this is an excellent example of the sheer flexibility that Git provides us with. See, if we can't "pull" the Ports repository (in)to our "full" directory we'll just present our "full" directory to the repository by manually "connecting" the two :-/

Sort off... First we made our own local Git repository by using git init. And as I mentioned earlier: you always use a full blown repository no matter what. It doesn't matter if you clone an existing repository or make a new one. And about that cloning... If you clone a repository you're effectively taking a few steps, steps which you can also do manually. First we fetched everything there is to know about the remote repository - but only that - by using git-fetch(1). This provided us with a lot of useful information about the remote, in specific Git learned about new branches and tags (as shown above).

The only remaining problem though were the two different branch names. If you look at the above output again you'll notice that our new local branch was called master but when fetching the information we see mention of remote/main, so the remote branch is called main (when in doubt you can try to look up HEAD using: git branch -r --contains @ (only works after the 'merge')).

So now we know that we have 2 separate branches: master (local) and origin/main (remote). As such, all that's left to do is to combine them, and we did that using git-merge(1). The only step remaining was to set an upstream, which is what git-branch(1) did for us.

Simply put: Git truly lives up to the Unix philosophy and some of its commands (like "pull" or "clone") are basically nothing more but a series of other commands which get executed in sequence.

And if you look at the differences between a cloned repository and a "fetched" one you'll notice that it's a complete non-issue:
Code:
peter@vps:/usr/ports $ git status
On branch main
Your branch is up to date with 'origin/main'.

peter@vps:/usr/ports $ git remote -v
origin  https://git.freebsd.org/ports.git (fetch)
origin  https://git.freebsd.org/ports.git (push)
peter@vps:/usr/ports $ git branch
* main
Here you see what it looks like when you simply clone the Ports repository. Notice the mention of origin/main up there? And after the second command you can clearly see that origin is a referral to https://git.freebsd.org. Since main is our local branch... it should go without reason that origin/main is simply a referral to the remote branch. This remote branch is "connected" to ours and therefor Git always compares the status of our local branch with that of the remote. In Git terminology we refer such a "connected" remote branch as an upstream.

Now let's compare this with the other methodology:
Code:
peter@vps:/home/peter/temp/myports $ git status
On branch master
Your branch is up to date with 'origin/main'.

peter@vps:/home/peter/temp/myports $ git remote -v
origin  https://git.freebsd.org/ports.git (fetch)
origin  https://git.freebsd.org/ports.git (push)
peter@vps:/home/peter/temp/myports $ git branch
* master
As you can see it's roughly the same; the main difference is that instead of using main we're using master as our local branch. Which is actually not a bad thing because it might make it easier on you to keep track of your local branch and the remote one(s).

Now, you could be tempted to simply (ab)use this information and make sure that all your new local branches are automatically called main. Problem solved, eh? Well... no. See, the name of a main branch is fully up to the person who set up the local repository. Mine always use 'master' because that's what I prefer. So by using the method I demonstrated above it doesn't matter what the remote branch is going to be named, you'll always be able to download & connect them, no matter what. Just remember to check up on HEAD (denoted by @) when in doubt.

End of Part I (message too long (as usual!) o_O)
 
OP
ShelLuser

ShelLuser

Son of Beastie

Reaction score: 2,054
Messages: 3,770

(Part II)

So what are the 'Git advantages'?


There are a lot of "gidvantages" here 😁 but of course it depends on circumstances if these will also apply to you.

Keeping track of local changes

Let's say for the sake of argument that I found a bug in devel/git, Not the software itself but the Makefile. Now, if it's a small issue I can simply apply my changes, create & e-mail the diff and be done with it. But what if this is going to take more time? Time during which I have to keep my ports up to date (that's important on a server) and to make things worse they already announced a big update to appear tomorrow! :eek:

Meaning? Well, if I simply start hacking then I can kiss my changes goodbye right after the ports collection got updated. Should I just postpone the updates then? I don't think so...
Code:
peter@vps:/usr/ports/devel/git $ git branch --no-track fixes
peter@vps:/usr/ports/devel/git $ git branch
  fixes
* main
peter@vps:/usr/ports/devel/git $ git switch fixes
Now I can hack to my hearts desire without having to worry about my changes suddenly getting overwritten by new updates. However, I do need to make sure that I either switch back to main before I update the ports collection or I need to specifically tell Git to apply the updates to the main branch.

So, either I use: vps:/usr/ports # git switch main && git pull, or: vps:/usr/ports # git pull origin main, optionally followed by git merge main to apply any updates from main to my fixes branch.

Oh, and also... no more messing with diff either I think:

Code:
peter@vps:/usr/ports/devel/git $ git diff Makefile
diff --git a/devel/git/Makefile b/devel/git/Makefile
index 3082a5968361..2a6fc3090ec5 100644
--- a/devel/git/Makefile
+++ b/devel/git/Makefile
@@ -11,7 +11,7 @@ DISTFILES=    ${DISTNAME}${EXTRACT_SUFX} \
EXTRACT_ONLY=  ${DISTNAME}${EXTRACT_SUFX} \
                ${PORTNAME}-manpages-${DISTVERSION}${EXTRACT_SUFX}

-MAINTAINER=    garga@FreeBSD.org
+MAINTAINER=    ShelLuser@users.noreply.github.com
COMMENT=       Distributed source code management tool ${COMMENT_${FLAVOR}}

LICENSE=       GPLv2
Time for a hostile takeover! :sssh: (with apologies to Garga (= real port maintainer), pls no kill me! :beer:).

Moving to/from binary packages

As you probably know a binary package (" # pkg install git") is nothing more but a port that got build using the default settings. There are 2 repository branches which you can track: quarterly and latest (see also /etc/pkg/FreeBSD.conf). As their name suggests 'latest' is the repository as it is now while quarterly is 'delayed' and gets released 4 times a year.

Now let's say for the sake of argument that we have a server which tracks quarterly, and we need to urgently re-configure ("customize") one of the packages. Now what?! I mean... sure: download ports collection, build port and then diving head first into dependency hell?! Nope!

Now we can resolve this problem very easily thanks to Git:
Code:
peter@vps:/usr/ports $ git branch -r | grep 202.
  origin/2020Q1
  origin/2020Q2
  origin/2020Q3
  origin/2020Q4
  origin/2021Q1
  origin/2021Q2
See what I mean? For every quarterly release FreeBSD made sure to provide a branch. So all we have to do is to switch our Ports collection to the branch we need: # git switch 2021Q1, then we simply install our customized port (and eventually upgrade to the latest release).

But there's more...
Code:
peter@vps:/usr/ports $ git branch -r | grep Q | head -2
  origin/2014Q1
  origin/2014Q2
Although fully unsupported it would allow you to restore an old(er) version of FreeBSD which has most likely long passed its EOL.

It gets even better...
Code:
peter@vps:/usr/ports $ git tag | grep eol
10-eol
4-eol
5-eol
6-eol
7-eol
8-eol
9-eol
peter@vps:/usr/ports $ git log 5-eol^!
commit a54fe1ca9c33931d7c4381a84ee11454f9831c08 (tag: 5-eol)
Author: cvs2svn <cvs2svn@FreeBSD.org>
Date:   Sat May 31 23:01:15 2008 +0000

    This commit was manufactured by cvs2svn to create tag 'RELEASE_5_EOL'.
Pretty cool, right?

Distributing your Ports collection

If you can log onto your server from another (remote) server you can also easily clone any repositories on it. Easiest (and safest) is to use SSH: # git clone ssh://peter@10.0.2.25:/usr/ports /usr/ports. This can save you plenty of bandwidth.

So what's the catch?
Code:
peter@vps:/usr/ports $ zfs list -r zroot/ports
NAME                    USED  AVAIL  REFER  MOUNTPOINT
zroot/ports            6.68G   105G  1.05G  /usr/ports
zroot/ports/distfiles  2.73G   105G  2.73G  /usr/ports/distfiles
zroot/ports/packages   2.89G   105G  2.89G  /usr/ports/packages
First a 1Gb increase in required storage space. zroot/ports used to take up around 5.6Gb on my server and as you can see this is now 6.68.

Another catch is the lack of /usr/ports/INDEX (see also portsnap(8)) which means that using pkg-version(8) will gobble up a lot more time due to lacking index files. Of course that can easily be remedied by running make index in /usr/ports.


The source tree

As you can probably imagine we can simply apply most of the things we've learned above to the source tree as well, programs like Git are quite consistent and fortunately for us the FreeBSD maintainers also made sure that their implementations follow this principle too.

So... all we basically have to do is replace ports.git with src.git and instead of using /usr/ports we're now using /usr/src.

First let's clone the source code: # git clone https://git.freebsd.org/src.git /usr/src.

If you have been using Subversion so far then I strongly suggest you remove all the contents of /usr/src and then check out a new repository as shown above. Careful that you don't accidentally remove your kernel configuration (most obvious location would be: /usr/src/sys/amd64/conf/ (assuming you're using a 64bit architecture)).

Directory not empty?

And remember: if this doesn't work because of system files, as will be the case with UFS filesystems, then you simply follow this procedure:
  1. Enter /usr/src: cd /usr/src.
  2. Make a new Git repository: git init.
  3. Add a 'pointer' to the main repository: git remote add origin https://git.freebsd.org/src.git.
  4. Fetch all the available data: git fetch origin.
  5. Merge both repositories: git merge origin/main.
  6. 'Connect' both repositories: git branch --set-upstream-to=origin/main.
Note that you can also use -u instead of --set-upstream-to= in #6, but I personally prefer the long version because then there can't be any doubt about what we're trying to do here.

How to use...

If you've maintained your FreeBSD server using the source tree before then you might have noticed something here... We didn't specify anything regarding versions, so say something like releng/12.2 or maybe releng/13.0. What gives?! Don't worry, it's all here. But because of that we have to apply a specific workflow to get the most of this.

First... start by checking out what we got here: git branch -r and be amazed:
Code:
peter@vps:/usr/src $ git branch -r

  origin/HEAD -> origin/main
  origin/main
  origin/releng/1
  origin/releng/10.0
  origin/releng/10.1
[...]
  origin/releng/12.1
  origin/releng/12.2
  origin/releng/13.0
See what I mean?

So the first thing we will need to do here is to point our repository to the version we're using. In my example that is 12.2, therefor we use the following command: git switch releng/12.2. Of course if you already use 13.0 you'd use: git switch releng/13.0, and so on.

Now pardon me for jumping briefly into my "fanboy mode" but... how cool is this?! :cool: Not only do we have the source code for a whole operating system at our disposal here, but the days where we had to mess around with /usr/src and (maybe) /usr/src.new (or simply removing all contents (and then forgetting and losing our kernel source configuration :oops: )) are long behind us.

One tree to rule them all and in the Unix bind them...

(/small vent) - Sorry but even after having used the source tree for more than 8 years so far I'm still fascinated by the IMO awesome mechanic 😎 And if you ask me Git only makes this experience better, but that's just my 2 cents.

But there's one thing to keep in mind. If you switch to a specific branch then older versions of Git may not automatically track the remote branch, which means that you cannot just use git pull to update your source tree. Instead you'd need to specify that you want to update the whole source tree: git pull origin main, after which you'd apply all the possible updates onto the current releng/12.2 branch by forcing a checkout: git checkout -f releng/12.2, this will prevent any local changes from getting stored somewhere.

However, this only applies with older Git versions. A good way to check is to run git status while you're on a specific branch:
Code:
peter@vps:/usr/src# git status
On branch releng/12.2
Your branch is up to date with 'origin/releng/12.2'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        sys/amd64/conf/VPS

nothing added to commit but untracked files present (use "git add" to track)
As you can see Git mentions that my branch is up to date with origin/releng/12.2 which means that I don't have to bother myself and can use git pull to keep this branch up to date.

And this also brings us to an important issue: the kernel configuration.

Managing your kernel configuration

As you probably know you can either use the GENERIC kernel configuration or set up one of your own. If you do then you need to specify this during the make process using the KERNCONF build variable (which is usually added to /etc/make.conf, see also make.conf(5)) but only after you added your configuration file in the proper sub-directory of course.

For example... let's assume we want to remove support for HyperV and we're using the amd64 architecture. As such we'll create: /usr/src/sys/amd64/conf/hyper which contains the following:
Code:
include GENERIC

ident hype-kernel

nodevice hyperv
So far, so good, but from here on Git will keep bugging us that we have an untracked file, you can see an example of this in my previous code section.

There are multiple ways to deal with this.

First we could ignore the file. Since .gitignore is already part of the repository our best option is to use a local config file. Edit .git/info/exclude and add the location of our file: sys/amd64/conf/hyper. Now we won't be getting any warnings, but on the down side our kernel configuration is now also completely unprotected. See also gitignore(5) for more information on ignoring files in Git.

We could also create a new branch and then use that to maintain our config file. This has the advantage that you can always go back to a specific version, but it also becomes quite tricky because you'd basically work with a file that has to be available in 2 branches at the same time. Doable, but tricky.

Now, if you're used to protecting your source tree by turning it readonly ( zfs set readonly=on zroot/src for the win!) then you might already have applied a better failsafe by keeping your kernel config file outside the source tree and using a symbolic link. In my opinion this is the most elegant solution, also because it allows you to keep the file under version control outside the main repository which will prevent any conflicts.

gidvantages
Sorry, I think it sounds kinda funny so I'm keeping it in 😁
  • Same with the Ports collection it is very easy to provide this for other servers, all you basically need is to make sure that you can access this server remotely through SSH.
  • No more messing around with new source trees (which was the case with Subversion). As soon as a new version becomes available it gets propagated within the repository itself. All you have to do is switch branches.
  • Kernel hacking has become easier than before; because you can keep all your additions separated from the main tree while still applying all the remote updates to your local environment (= local branch).
  • In addition to the above point: git diff <file> will easily produce a diff which can be used by the people who maintain the upstream (or you could create a pull-request of course).
Also worth mentioning... if space is tight and you don't want to download the full backlog then that's also possible, for example you could use: git clone --single-branch --branch releng/12.2 https://git.freebsd.org/src.git src.

But the difference in size might well surprise (or disappoint?) you:
Code:
peter@vps:/home/peter/temp $ du -hd1
2.4G    ./kernel12_2
2.5G    ./kernel.full
4.9G    .
peter@vps:/home/peter/temp $ zfs list zroot/src
NAME        USED  AVAIL  REFER  MOUNTPOINT
zroot/src  1.90G  99.9G  1.87G  /usr/src
Here I made 2 clones. As you might have guessed kernel.full contains the full backlog (so I cloned the whole src.git repository into this directory) whereas kernel12_2 contains a single branch. You can see it is smaller, but it doesn't make that much of a difference.

For comparisons sake I also added my zroot/src dataset which is also a full clone, however here I applied ZFS compression which also makes quite a difference (and without compromising responsiveness I might add).


The FreeBSD documentation

Last but certainly not least is the FreeBSD documentation. Have you ever read the FreeBSD handbook? Well, now we're going to grab its source code as well. I hope you might have guessed by now: the only difference is that we'll now be using doc.git and the repository directory will be /usr/doc.

So to clone the documentation repository you'd use: # git clone https://git.freebsd.org/doc.git /usr/doc.
And also as before, if you're having issues because of system files (which can happen when you're using an UFS filesystem) then you'd follow this procedure:
  • Create an empty repository in /usr/doc: git clone => git init. (that was an embarrasing mistake! :oops:).
  • Point it to the main repository: git remote add origin https://git.freebsd.org/doc.git.
  • Fetch all information from the remote repository: git fetch origin.
  • Merge the branches: git merge origin/main.
  • 'Link' your repository to the remote upstream: git branch --set-upstream-to=origin/main.
After this you'll have full access to the official FreeBSD documentation; handbook, other books and website alike. Please refer to /usr/doc/README if you want to know how to build the documentation yourself. As it turns out it no longer suffices to install text-proc/docproj which is a bit of a shame, but manageable.

You can also provide this repository to other servers on your network but I honestly wouldn't bother and instead install a webserver such as Apache to provide the HTML version to your network. This will also make it much easier to maintain the documentation for your network(s).


And there you have it...

I think this about covers the whole range of official FreeBSD Git repositories, how to install them, how you can use them and yah... I hope you enjoyed reading this and maybe (hopefully) you also learned a thing or two...

Change can be a rough thing to deal with, especially with an environment such as FreeBSD which hardly ever changes. Trust me, if this whole thing bothers you then I know exactly how you feel. I've been there myself. But having said that I also believe that change can be a good thing from time to time; it helps us to evolve and move ahead. Heck, if we never changed or evolved we'd all still be staring at a bunch a stone wheels! 😁

And yes, I truly believe that we gained a lot from the move to Git and I hope you can agree to some extend when looking at some of my shared examples above. But, I'll also be honest with you guys by stating once again that I am definitely biased here because I've been using Git for over 3 years now and I really took a liking to it.

In fact... I've taken this fascination so far that all my servers are fully under Git control and as a result I can always look back at all my configuration changes, even those which dated 2 - 3 years ago even though my backups only have a retention of 7 days. And the only thing you might notice is a .git file in some directories such as /etc and /usr/local/etc. Hardly noticeable yet very powerful.

But... that is for another time.

For now, I hope that this was useful for you some of you and thanks for reading!

---
Update(s): If you see a crossed out word (or section) followed by an italic word (or section) then that indicates a corrected section of text.
 
Last edited:

jb_fvwm2

Daemon

Reaction score: 204
Messages: 1,787

just curious, why is the command
invalid, and is there an equivalent git command,
for
Code:
git switch stable/12.2
??
 
OP
ShelLuser

ShelLuser

Son of Beastie

Reaction score: 2,054
Messages: 3,770

why is the command invalid, and is there an equivalent git command,
for git switch stable/12.2??
(I took the liberty to change your [code] to a [cmd] in the quotation above; this makes one liners a little easier to read).

Anyway, it's really quite simple: there is no stable/12.2 branch. Never assume these things and always check for yourself first. At the time of writing this is available to us:

Code:
peter@vps:/usr/src $ git branch -r | grep stable
  origin/stable/10
  origin/stable/11
  origin/stable/12
  origin/stable/13
  origin/stable/2.1
  origin/stable/2.2
  origin/stable/3
  origin/stable/4
  origin/stable/5
  origin/stable/6
  origin/stable/7
  origin/stable/8
  origin/stable/9
See? If you want to work with 12 stable you'd use: git switch stable/12 instead.

(edit)

You're probably aware but I can't help mention this anyway: bear in mind that STABLE does not indicate a stable FreeBSD release. Stable within FreeBSD refers to a "monitored" development branch. While it is more reliable than CURRENT (all experimental changes get added here) it's still experimental by nature. If you want a regular release you'd use releng instead.

I mention this because releng/12.2 does exist.
 

bobmc

Member

Reaction score: 13
Messages: 47

To GIT or not to GIT.

I previously installed RC13 from USB stick and included the Ports. I don't have a specific problem but I want to maintain a cohesive and correct system so I moved /usr/ports to /usr/tests and got a replacement via git.
So, to install a new (fresh) ports collection you'd use: # git clone https://git.freebsd.org/ports.git /usr/ports. Done!
So I see the original RC13 ports :
1a. does not have a ./git directory .. does that mean I can't do a git update?
2a. UPDATING file last comment or change was 2021-01-06
3a. many files have the $FreeBSD release 13 stamp

And the git version:
1b. comes with ./git directory
2b. UPDATING last entry 2021-04-20
2c. the $FreeBSD does not have a release suffix

So what are the implications? Should we skip the Ports when installing from the release image?
 
OP
ShelLuser

ShelLuser

Son of Beastie

Reaction score: 2,054
Messages: 3,770

So I see the original RC13 ports :
1a. does not have a ./git directory .. does that mean I can't do a git update?
Correct. There's no Git database so... nothing to do here.

So what are the implications? Should we skip the Ports when installing from the release image?
I would, yes. In fact, I never installed the Ports 'bundle' that came with the installer image but always relied on portsnap (so: portsnap extract). I don't know the current status of that command but from what I read the idea is to revive it and to get it working with Git as well.

But yeah, for what's it worth I always install the ports collection manually.
 

bobmc

Member

Reaction score: 13
Messages: 47

I used SVN on a previous job so I know how it works. I know a little bit about GIT but it can be scripted so I don't expect a problem. However, the problem now is I don't know which system to use for maintaining ports.
 
OP
ShelLuser

ShelLuser

Son of Beastie

Reaction score: 2,054
Messages: 3,770

Things can definitely be scripted quite easily. Yah, I understand the problem here... well, for what's it worth:


This is what I'm using.. it's a set of 3 shell scripts which I made to make things easier on me together with Portmaster. Some friends of mine really liked it so I figured I'd dump it on GitHub. Bear in mind that I'm not claiming that this is a high-tech setup or something, it's pretty crude around the (virtual) edges but still...

Maintenance for me consists of running refresh which updates the ports collection (using Git) and also checks for updated ports (and logs their names; it also tries to look them up in the changes database through use of pkg-updating(8) but that's not always very reliable). When done I then run update which actually updates the ports by building them using Portmaster.

The last script, pmmove is used when a port got moved for some reason. Portmaster can detect this and sort out the database for you, this script basically takes care of it.

Maybe these scripts can give you some ideas to build your own?
 
OP
ShelLuser

ShelLuser

Son of Beastie

Reaction score: 2,054
Messages: 3,770

I downloaded your -assist files and have a question. 'USAGE' says "clone the repository".

? https://github.com/freebsd/freebsd-ports/tree.. It's the link given by FreePorts ..is this the repository? Is it known as "upstream"?
Ah, good point. Updating the documentation is still on my famous todo list, I'll probably work on that some more this weekend. Yah, I see that the wording could ("should") be much more clear there: I'm referring to the GitHub project repository, so this link.

The idea behind this project is that the repository is a working setup in itself, so you clone the 'assist repository', you initialize things by running 'refresh' after which you can run 'update' to actually update stuff (= install any optional updated ports).
 

neilulrich

New Member

Reaction score: 1
Messages: 12

Mixed ports all over? dir, /usr/ports, zroot/ports
remove /usr/ports BUSY dataset does not exist. ended telling noobe to just use pkg
I got it well done!
 

neilulrich

New Member

Reaction score: 1
Messages: 12

noobe thought had to build git
Keep in mind
: mixing ports and binary packages is a very bad idea!
my bad for pointing to [Guide] made more confusion. noobe had portsnap already
 
OP
ShelLuser

ShelLuser

Son of Beastie

Reaction score: 2,054
Messages: 3,770

neilulrich, pkgs are stable but the source of ports are evolving.
What is that supposed to mean? Packages are build from ports.

Compiling ports will override pkgs so that your system is in an unknown state.
Errr, what?!

Compiling != installing.

noobe thought had to build git
Keep in mind
: mixing ports and binary packages is a very bad idea!
my bad for pointing to [Guide] made more confusion. noobe had portsnap already
Is this going anywhere? Because from where I'm standing you guys are merely posting nonsense on this thread.
 

bobmc

Member

Reaction score: 13
Messages: 47

What is that supposed to mean? Packages are build from ports.


Errr, what?!

Compiling != installing.


Is this going anywhere? Because from where I'm standing you guys are merely posting nonsense on this thread.
It seemed like neilulrich was overloaded with details. My explanation was bonkers but I supported the wisdom of not mixing ports and packages. He can look at the port software without compiling or installing.
 

bobmc

Member

Reaction score: 13
Messages: 47

Off-topic from Git,



Some of that old wisdom is no longer true. For people who'd like to continue with this thread, I suggest:

I read all the forum stories and the handbooks. Struggled to merge the multiple sources of information (sometimes stale or deprecated). Finally, I wrote a script which adopted a port. So I am making progress.
 
Top