Solved What is the mechanism to patch upstream sources in /usr/src/contrib/* ?

Mjölnir

Daemon

Reaction score: 1,502
Messages: 2,114

For ports(7), there's the files/patch-* that I can apply with make -C section/portname patch. There must be a similar mechanism to patch(1) the external sources pulled into the BeaSD under contrib/* to fit the BeaSD's demands. But how & where? Any hints to point me in the right direction is highly appreciated.
P.S.: Current urgent background: PR 252663, which seems to be already fixed upstream, but I'm just lacking the knowledge how to incorporate upstream tcsh(1) into my source tree. I did git-lite pull to shadow the native BeaSD's contrib/tcsh, but that fails, since it's not patched...
 

Zirias

Daemon

Reaction score: 1,421
Messages: 2,479

Well, I doubt there's a builtin mechanism for patch management, but maybe I'm wrong on this. I was interested in the "background" here (although I don't like C shells), but something is wrong :eek:
Code:
$ host bugs.freebsd.org
Host bugs.freebsd.org not found: 3(NXDOMAIN)
edit: ok, that's fixed now. Well, looking at the PR … such things in base take their time. Maybe use a shell from ports? ;) (If this is for root, there's a reason there is also a toor account, so one of them can keep a shell from base, for emergencies)
 
OP
Mjölnir

Mjölnir

Daemon

Reaction score: 1,502
Messages: 2,114

EDIT: So you're suggesting to cah^H`Hhange root's default shell? Also note tr^Hhe footnote
Thank you for your kind support, next one, please. root's default shell is csh(1) IIRC, and multiple VTs are default since decades, so IMHO this option set savehist = (x000 merge lock) is not esoteric humbaquatsch, but should be default anyway, right?
EDIT: DD^H^HEEI^H^HDR^iT: PLAINTEXT: EDIT: I do welcome that you're investing time to reflect on this, but changing root's shell is NPoD (not point of discussion). root's shell has to be rock solid by the yea sound spoken trueth of @any_reasonable_user!
 

PMc

Daemon

Reaction score: 671
Messages: 1,354

Not sure what this is actually about: patching the buildworld? changing root's shell? horrible typing mistakes?
A) Mjölnir: please don't use a gadget with touchscreen and spell-correction. It's just terrible.
B) there is no sacrifice-priest engaged to call down the wrath-of-the-gods(tm) if we change root's shell. There are a couple of good reasons not to do it, but then it is all at the discretion of the local admin.
C) I suppose the usr/src/contrib is treated just like all of usr/src in that regard. Patching that, is, well, an issue. You need to:
1. know how to come up with the patch, i.e. the actual diffs
2. a means to maintain and manage these diffs
3. a means to apply the diffs to the source, and re-merge them during each OS upgrade.

I ended up with a LOT of shellscript for doing 2. and 3. (alongside with my world+port build mechanisms).
Concerning 1., I don't think there is a general means for drop-in replacement of contributed sources. There may also be strange cross-relationships between various directories (things built from different directories than where the source is actually placed). Whenever I needed to do something in there, it usually was working myself thru the details of the code and the makefiles and figuring out how it plays together.
 
OP
Mjölnir

Mjölnir

Daemon

Reaction score: 1,502
Messages: 2,114

Not sure what this is actually about: patching the buildworld?
Yes. Patching the upstream tcsh(1) sources with the same patchset that the maintainer on the FreeBSD side applies when pulling them into $SRCTOP/contrib/. In effect, updating the world to use the newer tcsh-6.22.03 instead of 6.21.00.
A) Mjölnir: please don't use a gadget with touchscreen and spell-correction. It's just terrible.
That was intentionally to give a visual hint that I had more than one beer (otherwise, you can only guess 'cause maybe I write even more bullsh(1)t than usual).
B) there is no sacrifice-priest engaged to call down the wrath-of-the-gods(tm) if we change root's shell. There are a couple of good reasons not to do it, but then it is all at the discretion of the local admin.
Although I learned to arrange with & even like the (t)csh(1) & it's funny syntax, I'm tempted to follow Zirias' & olli@'s suggetion and try the shells/zsh with and log into the toor account more frequently.
C) I suppose the usr/src/contrib is treated just like all of usr/src in that regard. Patching that, is, well, an issue. You need to:
1. know how to come up with the patch, i.e. the actual diffs
I guess the solution is to produce the diffs myself from diff $SRCTOP/contrib/tcsh ~/Downloads/tcsh-6.21.00 and apply them to the tcsh-6.22.03. Should be straightforward.
2. a means to maintain and manage these diffs
As long they not too many, I can live with collecting them in a directory ~/Projects/FreeBSD/src/{,$VERSION/}/patches
3. a means to apply the diffs to the source, and re-merge them during each OS upgrade.
Again, as long it's only a few, just doing it manually is reasonable IMHO.
I ended up with a LOT of shellscript for doing 2. and 3. (alongside with my world+port build mechanisms).
Yes, if you have many custom patches, you want to apply some management measures, to keep an overview & not accidently forget vital topics.
Concerning 1., I don't think there is a general means for drop-in replacement of contributed sources. There may also be strange cross-relationships between various directories (things built from different directories than where the source is actually placed). Whenever I needed to do something in there, it usually was working myself thru the details of the code and the makefiles and figuring out how it plays together.
Yes... That was the whole thing about my post, because up to now I had only very few patches, and very rarely ;)
 
Last edited:

Zirias

Daemon

Reaction score: 1,421
Messages: 2,479

Although I learned to arrange with & even like the (t)csh(1) & it's funny syntax, I'm tempted to follow [FONT=monospace]Zirias[/FONT]' & [FONT=monospace]olli@[/FONT]'s suggetion and try the shells/zsh with and log into the toor account more frequently.
Nothing wrong with that, toor exists for that purpose. AFAIK, it's "common practice" to use toor for the non-base shell, but it would work just as well the other way around. Important is just having ONE superuser account that will still work when something in /usr/local got messed up…
 

Snurg

Daemon

Reaction score: 589
Messages: 1,349

That was intentionally to give a visual hint that I had more than one beer (otherwise, you can only guess 'cause maybe I write even more bullsh(1)t than usual).
You usually don't write BS, but recently probably many got curious how many spirits you had.

Isn't the idea with the unionfs applicable here?
I liked your idea of of having a few directories whose contents just "overload" what is in the original /usr/src/...
So you can patch, edit and verify the modified file(s) and then mount the unionfs for building, and afterwards just unmount it.
Probably much easier than dealing with a lot of custom build scripts.
 
OP
Mjölnir

Mjölnir

Daemon

Reaction score: 1,502
Messages: 2,114

I was in the mood to get drunken & didn't have the discipline to do that alone, watching some movie on TV or such, but somehow managed to stay online...

Yes, sure, I'm doing that, leaving the genuine sources untouched. The question is:
  • where can I find the patches applied by the FreeBSD maintainer when s/he imports some upstream genuine sources into $SRCTOP/contrib/*?
 

olli@

Daemon
Developer

Reaction score: 1,252
Messages: 1,140

Although I learned to arrange with & even like the (t)csh(1) & it's funny syntax, I'm tempted to follow Zirias' & olli@'s suggetion and try the shells/zsh with and log into the toor account more frequently.

Personally I recommend to not log into the root or toor accounts at all. The only time I actually log into the root account is when I have to use single-user mode because something went wrong during an update. And single-user mode uses /bin/sh by default (it ignores the shell entry in the passwd record).

For normal administrative work, I use su -m (actually I have an alias su="su -m"). Then you will get a root shell that uses the same shell as your normal user account. That is, when your normal user shell is zsh, then su -m will give you zsh as the root shell.

However, if you do it like that, you have to be careful with your shell profiles, because the root shell will be executed from your user home directory, and it will process your normal user shell profiles. You probably want to set up a few things differently, depending on whether your shell is started as normal user vs. as root. For example, this is a simplified snippet from my ~/.zshrc:
Code:
if (( EUID == 0 )); then
        export MAIL=/var/mail/root
        export HISTFILE=/root/.zsh.history
        export LESSHISTFILE=/root/.lesshst
        alias ls='ls -abG'
        umask 022
else
        export MAIL=/var/mail/$LOGNAME
        export HISTFILE=$HOME/.zsh.history
        export LESSHISTFILE=$HOME/.lesshst
        alias ls='ls -bG'
        umask 077
fi
If you prefer to use /root as the home directory for your root shells, and also use zsh shell profiles from there, but still want to use su -m, then put the following snippet at the beginning of your ~/.zshenv:
Code:
if (( EUID == 0 )); then
        export HOME=/root
        export ZDOTDIR=/root
        if [[ -f $ZDOTDIR/.zshenv ]]; then
                source $ZDOTDIR/.zshenv
        fi
        return
fi

Just for the record, these are the various profiles used by zsh (I have this as a comment at the top of my files):
Code:
#       ~/.zshenv       EVERY zsh
#       ~/.zprofile     LOGIN shell only
#       ~/.zshrc        INTERACTIVE shell only
#       ~/.zlogin       LOGIN shell only
#       ~/.zlogout      LOGIN shell only (upon exit)
The vast majority of my personal settings (aliases, functions, key bindings, completion etc.) is in ~/.zshrc so I have it for every interactive shell, whether it’s a login shell or not.

Sorry, I think I've digressed quite a bit … ;)
 
OP
Mjölnir

Mjölnir

Daemon

Reaction score: 1,502
Messages: 2,114

Concerning sh(1)ell profiles, I feel safe using sudo(8) & direnv(1) & let them handle that stuff.

We went OT... Can you also comment on my genuine topic, or is my presumed solution ok: to just create & reuse the respective diff(1)s manually?
 

olli@

Daemon
Developer

Reaction score: 1,252
Messages: 1,140

Concerning sh(1)ell profiles, I feel safe using sudo(8) & direnv(1) & let them handle that stuff.
I don’t like sudo, but that’s a different topic …

We went OT... Can you also comment on my genuine topic, or is my presumed solution ok: to just create & reuse the respective diff(1)s manually?
I have to admit I didn’t understand your initial question completely. What is BeaSD? Is this yet another distribution of FreeBSD? I haven’t heard of it yet.
 
OP
Mjölnir

Mjölnir

Daemon

Reaction score: 1,502
Messages: 2,114

No no, FreeBSD = tenderly nicknamed the BeaSD. I wanted to try if the newer tcsh-6.22.03 built with my FreeBSD 12.2-REL would fix that bug "tcsh hang when merging .history" (presumably = PR 252663). FreeBSD 12.2-REL has tcsh-6.21.00. But it is not obvious where I can find the necessary patches that are applied to the external sources under $SRCTOP/contrib. For the ports, there are the diff's in files/patch-xyz.... Where can I find the equivalent patches for the external sources incorporated into base under $SRCTOP/contrib?
 

memreflect

Well-Known Member

Reaction score: 220
Messages: 258

I don't know all of the commands that happen behind the scenes, but you could get mostly the same effect by following a simple procedure rather than trying to duplicate the exact workflow of the contributor/maintainer. Despite the intent to build only tcsh, the build may still take quite some time if you haven't built its dependencies already:
Code:
SRCTOP=/usr/src
upstream=tcsh.upstream

pax -w -f tcsh.tar "$SRCTOP"/contrib/tcsh
git clone git://github.com/tcsh-org/tcsh "$upstream"
git -C "$upstream" checkout ceccc7f
cp -R "$upstream"/* "$SRCTOP"/contrib/tcsh
make -C "$SRCTOP" SUBDIR_OVERRIDE=bin/csh all install

Sure, there are now paths like $SRCTOP/contrib/tcsh/win32 and $SRCTOP/contrib/tcsh/Announce that aren't in the "official" update, but since those are never used in $SRCTOP/bin/csh/Makefile, they're simply files that are unused for the build.

To revert to the original tcsh for whatever reason, you can just obliterate $SRCTOP/contrib/tcsh, restore the backup, and reinstall:
Code:
rm -r "$SRCTOP"/contrib/tcsh
pax -r -f tcsh.tar
make -C "$SRCTOP" SUBDIR_OVERRIDE=bin/csh all install
You can omit the make(1) line to simply restore the directory without reinstalling the original tcsh. This is useful if you're using SVN/Git to manage your SRCTOP, and you can't pull the latest changes due to local modifications that have not been committed/pushed. On the other hand, this means the next time you make installworld or freebsd-update(1) to a version without the history fixes, the fixed tcsh you had installed will be gone, and you'll need to copy from the upstream dir and reinstall it again.
 

olli@

Daemon
Developer

Reaction score: 1,252
Messages: 1,140

No no, FreeBSD = tenderly nicknamed the BeaSD. I wanted to try if the newer tcsh-6.22.03 built with my FreeBSD 12.2-REL would fix that bug "tcsh hang when merging .history" (presumably = PR 252663). FreeBSD 12.2-REL has tcsh-6.21.00. But it is not obvious where I can find the necessary patches that are applied to the external sources under $SRCTOP/contrib. For the ports, there are the diff's in files/patch-xyz.... Where can I find the equivalent patches for the external sources incorporated into base under $SRCTOP/contrib?
Well, tcsh 6.22.03 has recently been imported into the vendor/tcsh branch. It has then been merged to HEAD with this commit. A patch doesn’t exist directly, because it is implied by the merge. You can generate a patch yourself by creating a diff between the vendor branch and HEAD’s contrib/tcsh directory.
 
OP
Mjölnir

Mjölnir

Daemon

Reaction score: 1,502
Messages: 2,114

memreflect, that last point is not necessary, because I always keep the genuine source tree as-is, and build in my user's ~/Projects/FreeBSD/src/$version, where I mount the genuine source tree below my additions & changes (mount_unionfs(8)).

olli@ yes, that's what I thought:
I guess the solution is to produce the diffs myself from diff $SRCTOP/contrib/tcsh ~/Downloads/tcsh-6.21.00 and apply them to the tcsh-6.22.03. Should be straightforward.
 
OP
Mjölnir

Mjölnir

Daemon

Reaction score: 1,502
Messages: 2,114

So the whole point of this thread is:
  • the external, so-called vendor sources are incorporated into the FreeBSD source tree (under $SRCTOP/contrib) using the respective VCS tools (mostly git(1) nowadays), instead of downloading the bare tar(1) ball.
  • ⇒ i.e. in effect, only diff(1)s are applied on import; thus there's no need to apply patches, that has already been done once, long ago when the resp. external project's product was incorporated (the vendor sources imported) for the 1st time.
  • I didn't know that & refused to derive it myself, although it would have been obvious because I read that PR 252663 (obviously that WvE is me) before I posted here...
  • the patch(1) is in that commit.
Issue SOLVED.
 
Top