Useful scripts

[Erichans]

I used to do a loop over pkg prime-origins and use pkg annotate in the loop

Out of curiosity: can you post your previous (non-SQL) sh script? (termbin if you like/applicable)

 

[W.hâ/t]

Out of curiosity: can you post your previous (non-SQL) sh script? (termbin if you like/applicable)

Sure. https://termbin.com/keec/

 

[JohnK]

Sure. https://termbin.com/keec/

What a wonderful, thoroughly useful script! Thank you.

 

[VladiBG]

Print the number of failed logging attempts in postfix maillog sorted by top 10. Usually those attempts are blocked by fail2ban which by default search 10m in the past with retry attempts of 5 but some bad actors are connecting once every 20m from different IP like in the following example

awk -F":" '/authentication failed/ {print $5}' maillog | sort -n | uniq -c | sort -nr | head -10

58 unknown[45.144.212.130]
40 unknown[141.98.10.46]
37 unknown[81.30.107.89]
37 unknown[81.30.107.145]
37 unknown[81.30.107.121]
36 unknown[81.30.107.64]
36 unknown[81.30.107.29]
36 unknown[81.30.107.189]
36 unknown[81.30.107.174]
36 unknown[81.30.107.159]

That's why fail2ban need to be adjusted based on the number of attempts like this:

[postfix]
enabled = true
mode = extra
findtime = 60m
maxretry = 3

So it will search 60m in the past with maximum fail attempts of 3.

In my case after i block the entire subnet then I contact the abuse email address which can be found in RIPE database so the owner of this IP block can take some actions.

 

[Erichans]

I'm not sure if there's a simplier way to do this, [...]
EDIT:
pkg shell is likely a safer choice than sqlite3

pkg-shell(8) issues a stark warning:

DESCRIPTION
       pkg shell is used to interact with the local or remote database through
       a  sqlite3  console.  Extreme care should be taken when using this com-
       mand.

       Do not rely on this command.   The  underlying  schema  is  subject  to
       change  on any release.	Use pkg-query(8), pkg-rquery(8), pkg-search(8)
       and pkg-set(8) instead for querying and modifying the database.	 These
       commands	are expected to	have a stable API, unlike the database schema.

I tried your SQL script* and your shell script (a copy of your original shell script is also below**) but the latter gave me problems. With your shell script in my setup this resulted in:

[1-0] % poudlist-org-pkg.sh | grep gpu-firmware
graphics/gpu-firmware-intel-kmod@alderlake broxton cannonlake dg1 dg2 elkhartlake geminilake icelake kabylake rocketlake skylake tigerlake
[2-0] %

For my set up the listed flavors are wrong as not all of them are installed as automatic; furthermore this is an invalid specification of flavors, see poudriere(8) - FLAVORS. A detailed explanation is given below**.

Adapting the shell script is probably possible in this manner but then even more pkg-query(8) or pkg-annotate(8) calls are necessary. This would result in further speed penalties as going in and out of the pkg database for every relevant package is highly inefficient.

---

Given the unstable nature of the internal SQL schema of the pkg database as mentioned by pkg-shell(8) , I tried an alternative with a minimum of pkg db invocations. I started out with sed(1) and standard commnand line tools, that basically consist of a number of transformations without any programming:

#!/bin/sh
# Based on installed packages, create a list of ports (flavored and non-flavored) for a poudriere bulk build
pkg query -x -e '%a = 0 && %?A = 1' '%At="%Av"\t%o' '.*' \
| sed -E '/^flavor=/! s/.*\t(.*)/\1 \1/
          /^flavor=/  s/^flavor="([^"]*)"\t(.*)/\2@\1 \2/ ' \
| sort -r -k 1 \
| uniq -f 1 \
| cut -w -f 1

As an awk(1) solution that eliminates the use of other utilities and pipes, this is what I came up with:

#!/bin/sh
# Based on installed packages, create a list of ports (flavored and non-flavored) for a poudriere bulk build
pkg query -x -e '%a = 0 && %?A = 1' '%At %Av %o' '.*' \
| awk '
     FNR == 1 { origin = $3
                flavoredPort = ( $1 == "flavor" ? origin "@" $2 : "" )
              }
     FNR != 1 { if ( $3 != origin ) {
                    print ( flavoredPort ? flavoredPort : origin )
                    origin = $3
                    flavoredPort = ( $1 == "flavor" ? origin "@" $2 : "" )
                } else if ( $1 == "flavor" ) flavoredPort = origin "@" $2
              }
  END { print ( flavoredPort ? flavoredPort : origin ) } '

Some superficial timings suggest that the awk(1) script outperforms the sed(1)-utilities based script, at least in my limited package environment set up that results in a build list of 32 ports.

Both scripts rely on the fact that all packages have at least one Annotation entry. Current FreeBSD remote package repositories suggest that this is the case, most notably they all seem to have repository="FreeBSD". However, it may be that locally generated packages of an administered local package server may have 0 annotations if set up with that intent. To include installed packages that have no annotations at all, an additional pkg command has to be used: pkg query -x -e '%a = 0 && %?A = 0' '%o '.*'

___
* The SQL solution you presented earlier correctly generated the list with graphics/gpu-firmware-intel-kmod@kabylake in my setup:

[11-0] % poudlist-sql-pkg.sh | grep firmware
graphics/gpu-firmware-intel-kmod@kabylake
net/wifi-firmware-iwlwifi-kmod@22000
[12-0] %

**

Spoiler: Explanation

As individual package annotations can only be manipulated with pkg-annotate(8), it seems unfortunate that this aspect is not integrated in pkg-query(8). However, pkg-query(8) can be used to list all annotations. This is used in the two provided scripts.

With the original sh script the generated list can contain problamatic entries when several flavored packages have the same origin. Flavored packages of the same origin may be mutually exclusive (CONFLICTS_INSTALL), like for example with vim(1), /editors/vim:

Package flavors (<flavor>: <package>)

        console: vim
        gtk2: vim-gtk2
        gtk3: vim-gtk3
        motif: vim-motif
        x11: vim-x11
        tiny: vim-tiny

However, it may also happen that flavors can coexist as installed packages when having the same origin, graphics/gpu-firmware-intel-kmod:

Package flavors (<flavor>: <package>)

        skylake: gpu-firmware-intel-kmod-skylake
        broxton: gpu-firmware-intel-kmod-broxton
        kabylake: gpu-firmware-intel-kmod-kabylake
   [...]

With the original shell script in my setup this resulted in:

[1-0] % cat poudlist-org-pkg.sh | nl
     1  #!/bin/sh

     2  list=
     3  for origin in $(pkg prime-origins)
     4  do
     5          flavor=$(pkg annotate -qS ${origin} flavor)
     6          [ -n "${flavor}" ] && origin=${origin}@${flavor}
     7          echo ${origin}
     8  done
[2-0] % poudlist-org-pkg.sh | grep gpu-firmware
graphics/gpu-firmware-intel-kmod@alderlake broxton cannonlake dg1 dg2 elkhartlake geminilake icelake kabylake rocketlake skylake tigerlake

The problem lies in the fact that $(pkg prime-origins) in line 3 lists all packages that are constraint by '%a = 0', as its definition shows:

% pkg alias prime-origins
prime-origins        'query -e '%a = 0' '%o''

However, ${origin}@${flavor} in line 6 is used without the restriction of %a = 0'. Thus, the original sh script generates a list of all flavors for the specified ${origin}, irrespective of a particular flavored package was installed automatically or not. In my set up there happened to one flavored packages (gpu-firmware-intel-kmod-kabylake-20230625.1402000) that was reported not to be automatically installed; all other flavors for that origin however were automatically installed, for example:

[10-0] % pkg query -x  '(automatically_installed=%a) %o %n' 'kabylake|alderlake'
(automatically_installed=1) graphics/gpu-firmware-intel-kmod gpu-firmware-intel-kmod-alderlake
(automatically_installed=0) graphics/gpu-firmware-intel-kmod gpu-firmware-intel-kmod-kabylake
[11-0] %

The use of evaluation code %?A = 1 isn't strictly necessary, however it may serve to emphasize that it is necessary to produce any output given the '%At %Av %o' as query format. For a specified <pattern> when dealing with packages that have no annotations at all this strangely blocks any output. For testing, I used pkg-annotate(8) to delete all annotations from sysutils/lsof.

[0-0] % pkg inf lsof
lsof-4.99.4_2,8
Name           : lsof
Version        : 4.99.4_2,8
Installed on   : Mon Mar 31 20:27:35 2025 CEST
Origin         : sysutils/lsof
Architecture   : FreeBSD:14:amd64
Prefix         : /usr/local
Categories     : sysutils
Licenses       : lsof
Maintainer     : ler@FreeBSD.org
WWW            : https://github.com/lsof-org/lsof/
Comment        : Lists information about open files (similar to fstat(1))
Shared Libs required:
        libc.so.7
        libkvm.so.7
        libutil.so.9
Annotations    :
Flat size      : 236KiB
Description    :
Lsof (LiSt Open Files) lists information about files that are open by the
running processes.  An open file may be a regular file, a directory, a block
special file, a character special file, an executing text reference, a library,
a stream or a network file (Internet socket, NFS file or Unix domain socket).

See also fstat(1) in the base system.
[1-0] % pkg query -x -e '%a = 0 && %?A = 0' '%At %Av %o' 'lsof'
[2-0] % pkg query -x -e '%a = 0 && %?A = 0' '%o' 'lsof'
sysutils/lsof

As a final note: your SQL-based script nicely lists even those packages that have 0 (here: the stripped-of-Annotations-version of lsof) Annotations as compared to the above provided awk(1) solution:

[0-0] % poudlist-sql-pkg.sh | egrep '(lsof|gsed|gpu-firmware-intel-kmod)'
graphics/gpu-firmware-intel-kmod@kabylake
sysutils/lsof
textproc/gsed
[1-0] % poudlist-awk.sh | egrep '(lsof|gsed|gpu-firmware-intel-kmod)'
graphics/gpu-firmware-intel-kmod@kabylake
textproc/gsed
[2-0] %

 

[W.hâ/t]

Nice find with the awk(1) script. It leaves room for tweaks, like remove py* flavors to keep building FreeBSD's default version. I'll have a look at it later, I kinda don't understand the `FNR` lines...

Maybe post it in a single separate post. I'll edit/gray mine after with a link to yours.

 

[Espionage724]

Given the unstable nature of the internal SQL schema of the pkg database as mentioned by pkg-shell(8) , I tried an alternative with a minimum of pkg db invocations. I started out with sed(1) and standard commnand line tools, that basically consist of a number of transformations without any programming:

That's impressive, but I'm curious how that would be re-usable? I don't know what 2/3rds of pkg query -x -e '%a = 0 && %?A = 1' '%At="%Av"\t%o' '.*' is

I see random stuff like that in scripts online, and there's no way everyone is understanding what that stuff is before passing it behind a curl .sh. That line looks cool, but what if you wanted to use it in another script just slightly differently? Or would you re-create the whole command?

The wildest thing I came up with was this; I knew what all those flags did at the time, it used to work, but lmao I think -a 'a' was something weird like allow all cookies but good luck with the rest + different casing (iirc the flags also differed Linux/FreeBSD)

GDK_BACKEND='x11' surf -a 'a' -b -D -f -g -I -k -m -n -S -t -x 'http://localhost:8888'

 

[Erichans]

I kinda don't understand the `FNR` lines...

     1  #!/bin/sh
     2  # Based on installed packages, create a list of ports (flavored and non-flavored) for a poudriere bulk build
     3  pkg query -x -e '%a = 0 && %?A = 1' '%At %Av %o' '.*' \
     4  | awk '
     5       FNR == 1 { origin = $3
     6                  flavoredPort = ( $1 == "flavor" ? origin "@" $2 : "" )
     7                }
     8       FNR != 1 { if ( $3 != origin ) {
     9                      print ( flavoredPort ? flavoredPort : origin )
    10                      origin = $3
    11                      flavoredPort = ( $1 == "flavor" ? origin "@" $2 : "" )
    12                  } else if ( $1 == "flavor" ) flavoredPort = origin "@" $2
    13                }
    14    END { print ( flavoredPort ? flavoredPort : origin ) } '

Are you referring to just FNR == 1 (line 5) and FNR != 1 (line 8), or also to the code between { ... } in lines 5-13?

 

[W.hâ/t]

Are you referring to just FNR == 1 (line 5) and FNR != 1 (line 8), or also to the code between { ... } in lines 5-13?

I didn't understand the code. After playing around a bit, I understand what you did.

Is it important to keep the FNR tests? I revisited your code a bit, but I removed the FNR tests. I also added a function to select the flavor.

function select_flavor(f) {
    if (match(f, "py|qt"))
        return ""
    return f
}
{
    if ($3 == origin) {
        if ($1 == "flavor")
            flavor = select_flavor($2)
    } else {
        if (origin)
            print (flavor ? origin"@"flavor : origin)
        origin = $3
        flavor = ""
        if ($1 == "flavor")
            flavor = select_flavor($2)
    }
}
END { print (flavor ? origin"@"flavor : origin) }

select_flavor() can be expanded to take the origin as an argument and fit someone's need for more complex testing.

Thanks man!

 

[Erichans]

I'm curious how that would be re-usable? I don't know what 2/3rds of pkg query -x -e '%a = 0 && %?A = 1' '%At="%Av"\t%o' '.*' is

I see random stuff like that in scripts online, and there's no way everyone is understanding what that stuff is before passing it behind a curl .sh. That line looks cool, but what if you wanted to use it in another script just slightly differently? Or would you re-create the whole command?

Well, given "I don't know what 2/3rds [...] is" I'd say, you are well on your way and have travelled the first 1/3rd of the way

When there's 'simple' programming used in these scripts; you can rely on the man pages, like sh(1), sed(1) and awk(1) for the base and perhaps others in ports. Man pages however, are mainly for reference purposes. For learning purposes, I've found the many tutorials by Bruce Barnett very helpful:

• sh - the POSIX Shell
• sed
• awk

I also like The AWK Programming Language - 2nd edition

However, where no 'programming-of-sorts' is concerned, the ultimate references are the man pages, case in point: pkg-query(8). I was fortunate that I had two different scripts, the sql version and the sh-script version, that I could run against each other: compare the (different) output and try to hunt down the problem of the differences. As I explained in message #405 ("Adapting the shell script is probably possible [...] [but] highly inefficient.").

For writing the sed and awk based script I started with the original SQL script (called by poudlist-sql-pkg.sh below) and of the original sh script (see the 'Spoiler: Explanation' part in message #405 ) the use of pkg prime-origins and use its non-aliased format, then extending that to the regex format; then limiting the packages selected and then noticing the possibility of 'dumping' all annotations:

[0-0] % poudlist-sql-pkg.sh | grep -E '(vim|gsed)'
editors/vim@console
textproc/gsed
[1-0] % pkg alias prime-origins
prime-origins        'query -e '%a = 0' '%o''
[2-0] % pkg query -e '%a = 0' '%o'
  [ very long list ]
[3-0] % pkg query -x -e '%a = 0' '%o' '.*'
  [ very long list ]
[4-0] % pkg query -x -e '%a = 0' '%o' '(gsed|vim)'
textproc/gsed
editors/vim
[5-0] % pkg query -x -e '%a = 0' '%At=%Av %o' '(gsed|vim)'
FreeBSD_version=1401000 textproc/gsed
build_timestamp=2025-02-27T01:11:15+0000 textproc/gsed
built_by=poudriere-git-3.4.2 textproc/gsed
cpe=cpe:2.3:a:gnu:sed:4.9:::::freebsd14:x64 textproc/gsed
port_checkout_unclean=no textproc/gsed
port_git_hash=1bb147345 textproc/gsed
ports_top_checkout_unclean=no textproc/gsed
ports_top_git_hash=6ec2dc26d textproc/gsed
repo_type=binary textproc/gsed
repository=FreeBSD textproc/gsed
FreeBSD_version=1402000 editors/vim
build_timestamp=2025-04-24T07:30:03+0000 editors/vim
built_by=poudriere-git-3.4.2-11-gfa886a3d editors/vim
cpe=cpe:2.3:a:vim:vim:9.1:::::freebsd14:x64 editors/vim
flavor=console editors/vim
port_checkout_unclean=no editors/vim
port_git_hash=9d7c8ff2cb editors/vim
ports_top_checkout_unclean=no editors/vim
ports_top_git_hash=4de34b9860 editors/vim
repo_type=binary editors/vim
repository=FreeBSD editors/vim
[6-0] %

Except for the simplest of one-liners, for most, developing these scripts isn't a simply case of putting one's thought into an editor, running it and success is there . For me it involves reading trying stuff, thinking about stuff, trying more stuff etc.; I imagine that this process wasn't altogether that different for W.hâ/t while developing his sql and sh script.

If you can't understand certain details of a command (or script), try making a small cycle of:

1. Take a version of the script that produces output and superficially seems to do something; minimally: no syntax errors.
2. If you don't fully understand it, delete bits, until something simple remains; have documentation on hand.
3. Repeat cycle until the script does what you want it to do and you understand it.

The use of commands on the command line are by design, one could say: by necessity, terse and short. If you don't know certain options: read the man pages; that holds true for me and a lot of others.

Re-usable is what you make of it: how well you understand its individual components & its function, and can adapt the script to your circumstances and liking if necessary. For me, it's a learning process, sometimes quick and easy, other times not so.

I look at these scripts also to get ideas, about the problems they intend to solve, but also in what way they solve it and with what tools.
These scripts here are what your fellow users have come up with, not always perfect, but for them useful and hopefully usable for others, in exactly the form as given or with some modifications of your own. Remember they cannot know in advance every one's context, nor are they library developers!

Finally, after you've tried and have not succeeded, there's always this forum: no reason to despair. Lots of people have asked questions about sh(1) programming, quoting rules and specific command options.

 

[Erichans]

Is it important to keep the FNR tests? I revisited your code a bit, but I removed the FNR tests. I also added a function to select the flavor.

That looks like you have achieved the integration of the two patterns and does not need the use of FNR anymore; personally I find the control flow harder to follow, but that may well be personal preference.

There are other ways to eliminate explicit 'line-numbering' tests for the two patterns; generally this enables some form of initialisation or pre-processing before the 'main loop' of pattern-actions start. You can either make clever use of the next statement*, or use the getline function in the BEGIN pattern, the latter results in not needing any line number tests (like with FNR) and eliminates the test for an 'empty' origin:

#!/bin/sh
# Based on installed packages, create a list of ports (flavored and non-flavored) for a poudriere bulk build
pkg query -x -e '%a = 0 && %?A = 1' '%At %Av %o' '.*' \
| awk '
BEGIN { # BEGIN pattern gets & processes first line as initialisation; next pattern starts at line/record #2
    getline
    origin = $3
    flavor = ( $1 == "flavor" ? $2 : "" )
}
{ if ( $3 != origin ) {
     print ( flavor ? origin "@" flavor : origin )
     origin = $3
     flavor = ( $1 == "flavor" ? $2 : "" )
  } else if ( $1 == "flavor" ) flavor = $2
}
END { print ( flavor ? origin "@" flavor : origin ) } '     **

___
*

#!/bin/sh
# Based on installed packages, create a list of ports (flavored and non-flavored) for a poudriere bulk build
pkg query -x -e '%a = 0 && %?A = 1' '%At %Av %o' '.*' \
| awk ' \
FNR == 1 { origin = $3
           flavor = ( $1 == "flavor" ? $2 : "" )
           next
         }
         { if ( $3 != origin ) {
               print ( flavor ? origin "@" flavor : origin )
               origin = $3
               flavor = ( $1 == "flavor" ? $2 : "" )
           } else if ( $1 == "flavor" ) flavor = $2
         }
END { print ( flavor ? origin "@" flavor : origin ) } '

** Edit: error amended:

-: END { print ( flavor ? origin "@" $flavor : origin ) } '
+: END { print ( flavor ? origin "@" flavor : origin ) } '

 

[W.hâ/t]

That looks like you have achieved the integration of the two patterns and does not need the use of FNR anymore; personally I find the control flow harder to follow, but that may well be personal preference.

Yes, it's a personnal preference. I found that FNR didn't belong there. I like the BEGIN version better though. The initialisation is explicit.

I though about optimizing here and there, but it's irrelevant, you wouldn't notice it. Also, I think it would decrease readability.

FNR == 1 { origin = $3
  flavor = ( $1 == "flavor" ? $2 : "" )
  next
}

From what I understand, this will be tested each line, and therefore does not optimize. Am I missing something?

 

[cy@]

People want useful scripts? Read the O'Reily book UNIX Power Tools. It's full of useful scripts.

 

[W.hâ/t]

There's a bug here: if ( $3 != origin )

Since we test if origin differs from the previous line and pkg-query(8) sorts its output (probably by package name), some port built with multiple flavors will be truncated to one origin.

I chose to test for built_by which comes before flavor and seems to be annotated by poudriere to every package it builds.

#!/bin/sh

match_flavor='
function select_flavor(f) {
    if (match(f, "^py|^qt"))
        return ""
    return "@"f
}
{
    if ($1 == "built_by") {
        print origin
        origin = $3
    } else if ($1 == "flavor" )
        origin = origin select_flavor($2)
}
END { print origin }
'
${@:-eval} "pkg query -e '%a = 0' '%At %Av %o'" | awk "${match_flavor}" | sort

$ sh mkplist.sh
$ sh mkplist.sh ssh admin@server PKG_DBDIR=/jail/var/db/pkg

 

[Erichans]

From what I understand, this will be tested each line, and therefore does not optimize. Am I missing something?

Yes, of the first pattern containing next the test is executed for every record/line. When reading the first line this test is executed and, as it evaluates to true, the action is performed; for all other lines the test evaluates to false and as a consequence the action is not performed. However, the second test: FNR != 1 is eliminated; so one out of two, for that version. I don't think that will result in a relevant difference in execution. Along the same lines about optimization as you noted.

With awk(1)* you get:

1. Automatic record/line handling from the first to the last line
2. Automatic and settable field seperator and field splitting.
3. Extended regular expression handling.
4. Consise, yet very readable scripting; usually one liners can be quite effective. Reasonable well equipped with programming constructs for a scripting language. May not be very fast in some aspects.
5. UTF-8 support
6. Good environment support; the pkg-query command can be called from within the awk script.

Of course you can program the entire script after the pkg-query in sh(1). Then you'd have to write out the outer loop over the pattern-action statements explicitly and grab together all the other awk built-in facilities. I'm not sure if that would execute noticable faster. What probably would execute faster, scripting wise, would be writing this in python, also because its run-time optimization. It would, however, probably not be as consise as awk script.

___
* sed(1) will get you #1; #3; @#4: very consise: terse and hardly any programming construct: flow control is very very rudimentary, state (= variables) is all but absent--you only have the hold space; #5. You can observe some of the differences between awk(1) and sed(1) in their respective scripts for this problem

 

[Erichans]

There's a bug here: if ( $3 != origin )

Since we test if origin differs from the previous line and pkg-query(8) sorts its output (probably by package name), some port built with multiple flavors will be truncated to one origin.

I chose to test for built_by which comes before flavor and seems to be annotated by poudriere to every package it builds.

#!/bin/sh

match_flavor='
function select_flavor(f) {
    if (match(f, "^py|^qt"))
        return ""
    return "@"f
}
{
    if ($1 == "built_by") {
        print origin
        origin = $3
    } else if ($1 == "flavor" )
        origin = origin select_flavor($2)
}
END { print origin }
'
${@:-eval} "pkg query -e '%a = 0' '%At %Av %o'" | awk "${match_flavor}" | sort

$ sh mkplist.sh
$ sh mkplist.sh ssh admin@server PKG_DBDIR=/jail/var/db/pkg

Good catch. This made me realise a fundamental shortcoming in my thinking about the problem so far. Addressing this I changed my script for the generation of a poudriere list where, as far as annotations are concerned, the decision regarding the flavor is based on the presence or absence of the flavor annotation, and not on any other annotation; more on that below. There's also a typo in my previous script above (amended).

The basic problem we are faced with is that a SQL program, such as the one you wrote is the best solution to be had, were it not for the fact that it relies on the internal structure of the package database. I can certainly understand that from a development and maintenance point of view one would reserve the right to make any changes at any moment that are deemed necessary. With the massive changes in pkg development ongoing, and its accompanying rapidly succeeeding versions of late I can imagine this to be relevant even more.

As the SQL path is not a viable reliable option for some duration the developers of pkg had to invent some small 'language' (very small in comparison to SQL) to address this issue. A full implementation would mean basically an implementation of some equivalent form of SQL, that is of course not realistic.

Currently, for installed packages that leaves us mainly with pkg-query(8) and pkg-annotate(8). For the problem on hand I cannot resist mentioning that only something like:
pkg query -x -e '%a == 0 && %At == flavor' '%At=%Av %o' '.*'
would have helped a lot. Now we are faced with a dump of all annotations that we have to process to achieve our goal.

The error you found made me realise that some minor reservations I had were not thought through enough.

Spoiler: testing details

When having installed individually of the following flavors:

• green-sardine of graphics/gpu-firmware-amd-kmod
• tahiti of graphics/gpu-firmware-amd-kmod
• verde of graphics/gpu-firmware-radeon-kmod

for testing purposes, you get:

[1-0] % pkg query -x -e '%a = 0 && %?A = 1' '%At %Av %o' '(green-sardine|verde)' | column -t
FreeBSD_version             1402000                           graphics/gpu-firmware-amd-kmod
build_timestamp             2025-04-24T07:46:24+0000          graphics/gpu-firmware-amd-kmod
built_by                    poudriere-git-3.4.2-11-gfa886a3d  graphics/gpu-firmware-amd-kmod
flavor                      green_sardine                     graphics/gpu-firmware-amd-kmod
port_checkout_unclean       no                                graphics/gpu-firmware-amd-kmod
port_git_hash               1bb147345d                        graphics/gpu-firmware-amd-kmod
ports_top_checkout_unclean  no                                graphics/gpu-firmware-amd-kmod
ports_top_git_hash          4de34b9860                        graphics/gpu-firmware-amd-kmod
repo_type                   binary                            graphics/gpu-firmware-amd-kmod
repository                  FreeBSD                           graphics/gpu-firmware-amd-kmod
FreeBSD_version             1402000                           graphics/gpu-firmware-radeon-kmod
build_timestamp             2025-04-24T07:52:26+0000          graphics/gpu-firmware-radeon-kmod
built_by                    poudriere-git-3.4.2-11-gfa886a3d  graphics/gpu-firmware-radeon-kmod
flavor                      verde                             graphics/gpu-firmware-radeon-kmod
port_checkout_unclean       no                                graphics/gpu-firmware-radeon-kmod
port_git_hash               1bb147345d                        graphics/gpu-firmware-radeon-kmod
ports_top_checkout_unclean  no                                graphics/gpu-firmware-radeon-kmod
ports_top_git_hash          4de34b9860                        graphics/gpu-firmware-radeon-kmod
repo_type                   binary                            graphics/gpu-firmware-radeon-kmod
repository                  FreeBSD                           graphics/gpu-firmware-radeon-kmod

I briefly thought about when the lines of each got mixed somehow, that would give wrong results; I dismissed that because it would make no sense for pkg-query(8) to produce such mangled output.

However, with two sequences in direct succession from the same origin, you would get:

[1-0] % pkg query -x -e '%a = 0 && %?A = 1' '%At %Av %o' '(tahiti|verde)' | column -t
FreeBSD_version             1402000                           graphics/gpu-firmware-radeon-kmod
build_timestamp             2025-04-24T07:46:49+0000          graphics/gpu-firmware-radeon-kmod
built_by                    poudriere-git-3.4.2-11-gfa886a3d  graphics/gpu-firmware-radeon-kmod
flavor                      tahiti                            graphics/gpu-firmware-radeon-kmod
port_checkout_unclean       no                                graphics/gpu-firmware-radeon-kmod
port_git_hash               1bb147345d                        graphics/gpu-firmware-radeon-kmod
ports_top_checkout_unclean  no                                graphics/gpu-firmware-radeon-kmod
ports_top_git_hash          4de34b9860                        graphics/gpu-firmware-radeon-kmod
repo_type                   binary                            graphics/gpu-firmware-radeon-kmod
repository                  FreeBSD                           graphics/gpu-firmware-radeon-kmod
FreeBSD_version             1402000                           graphics/gpu-firmware-radeon-kmod
build_timestamp             2025-04-24T07:52:26+0000          graphics/gpu-firmware-radeon-kmod
built_by                    poudriere-git-3.4.2-11-gfa886a3d  graphics/gpu-firmware-radeon-kmod
flavor                      verde                             graphics/gpu-firmware-radeon-kmod
port_checkout_unclean       no                                graphics/gpu-firmware-radeon-kmod
port_git_hash               1bb147345d                        graphics/gpu-firmware-radeon-kmod
ports_top_checkout_unclean  no                                graphics/gpu-firmware-radeon-kmod
ports_top_git_hash          4de34b9860                        graphics/gpu-firmware-radeon-kmod
repo_type                   binary                            graphics/gpu-firmware-radeon-kmod
repository                  FreeBSD                           graphics/gpu-firmware-radeon-kmod

This fails indeed with the script of message #411.

Your solution tackles this problem by using the built_by tag. Given the pkg-query(8) output ordening it correctly identifies each transition, even with the same origin. However, there are a few limitations/problems. Your script does not initialise the origin variable, as a consequence it (used with pkg query -x -e '%a = 0' '%At %Av %o' '(yellow|green)') outputs an empty first line:

[1-0] % mkplist.sh | nl -b a
     1
     2  graphics/gpu-firmware-amd-kmod@green_sardine
     3  graphics/gpu-firmware-amd-kmod@yellow_carp
[2-0] %

The poudriere-bulk description:

-f file
Build ports listed in the file.
The path to the file has to be absolute. Ports must be specified in the form of “category/port” 
and sh(1)-style comments are allowed. Multiple -f file arguments may be specified at once.

Although perhaps not an issue for poudriere (at the moment), empty lines are not mentioned; if they are ignored at the moment that may change at some point the future; at least a precise description of the handling of empty lines is not mentioned. This could be alleviated by adding a BEGIN pattern as indicated, or by changing the print statement (line 12) into the version on line 11 as shown below:

     1  #!/bin/sh
     2  match_flavor='
     3  function select_flavor(f) {
     4      if (match(f, "^py|^qt"))
     5          return ""
     6      return "@"f
     7  }
     8  #BEGIN { origin = "#" }
     9  {
    10      if ($1 == "built_by") {
    11          #if ( origin ) print origin
    12          print origin
    13          origin = $3
    14      } else if ($1 == "flavor" )
    15          origin = origin select_flavor($2)
    16  }
    17  END { print origin }
    18  '
    19  ${@:-eval} "pkg query -e '%a = 0' '%At %Av %o'" | awk "${match_flavor}" | sort

As you've mentioned the built_by annotation is output before the flavor annotation: this is essential. If the built_by annotation was output after the flavor annotation the script would fail in producing a correct result. The current output is probably sorted by annotation tag by default or by an explicit SQL sort. I would feel more secure if at least pkg-query(8) would have mentioned at its Multiline patterns sub heading something like that these particular lines are sorted but that is not mentioned. More importantly the built_by annotation must always be present, otherwise the transition would not be noticed correctly. However, when building a port locally the built_by annotation is not generated, sysutils/lsof build locally results in:

[1-0] % pkg info -A lsof
lsof-4.99.4_2,8:
        FreeBSD_version: 1402000
        cpe            : cpe:2.3:a:lsof_project:lsof:4.99.4:::::freebsd14:x64:2
[2-0] %

What is needed is a good way to identify a transition. What is missing is a way to uniquely identify each package: the missing information! Adding the package name results in:

#!/bin/sh
# Based on installed packages, create a list of ports (flavored and non-flavored) for a poudriere bulk build
pkg query -x -e '%a = 0 && %?A = 1' '%At %Av %o %n' '.*' \
| awk '
BEGIN { #BEGIN pattern gets & processes first line as initialisation; next pattern starts at line 2
        getline
        origin = $3
        pkgname = $4
        flavor = ( $1 == "flavor" ? $2 : "" )
      }
{ if ( $4 != pkgname ) {
     print ( flavor ? origin "@" flavor : origin )
     origin = $3
     pkgname = $4
     flavor = ( $1 == "flavor" ? $2 : "" )
  } else if ( $1 == "flavor" ) flavor = $2
}
END { print ( flavor ? origin "@" flavor : origin ) } '

For testing purposes one could even extend the last field to %n-%o and randomly 'shuffle' the entire pkg-query(8) output using shuf(1) from sysutils/shuf *
___
*

Spoiler: shuf(1) testing

Using mkpl-4f.sh (the new script in this message) and mkpl-4f-shuf.sh:

#!/bin/sh
# Based on installed packages, create a list of ports (flavored and non-flavored) for a poudriere bulk build
pkg query -x -e '%a = 0 && %?A = 1' '%At %Av %o %n-%o' '.*' \
| shuf \
| sort -k 4 \
| awk '
BEGIN { #BEGIN pattern gets & processes first line as initialisation; next pattern starts at line 2
        getline
        origin = $3
        pkgname = $4
        flavor = ( $1 == "flavor" ? $2 : "" )
      }
{ if ( $4 != pkgname ) {
     print ( flavor ? origin "@" flavor : origin )
     origin = $3
     pkgname = $4
     flavor = ( $1 == "flavor" ? $2 : "" )
  } else if ( $1 == "flavor" ) flavor = $2
}
END { print ( flavor ? origin "@" flavor : origin ) } '

[1-0] % mkpl-4f.sh | sort > plist.txt
[2-0] % mkpl-4f-shuf.sh  | sort > plist-shuf.txt
[3-0] % diff plist.txt plist-shuf.txt
[4-0] %

 

[Steen E]

Hi there

I have looked around in /usr/src/secure/caroot, just out of curiosity, and started to write a shell script mimicking the 'MAca-bunde.pl'. Not so surprising, but it turns out to be difficult!

I do not understand mathematics, which is otherwise always a good help - I mean, to understand mathematics would be a good help - and don't know how to convert the octets of SERIAL_NUMBER into decimal. The octets does not seem to follow the rule of 8^(digit-position). For example, one of the QuoVadis certificates has serial 1289 in decimal, parenthesized as (0x509), which is quite funny, and then a series of octet that should hold the same value: 002 002 005 011. That is 2*(8^3) + 2*(8^2) + 5*(8^1) + 11*(8^0), which is 1201?

As of now, the script can split the nss bundle - the formatting is really done by calling openssl(1ossl) - but sorting is not working, that is to say, the result does not correlate with the sorted certificates found in the subdirs of /usr/src/secure/caroot. For example, "Certum_Root_CA.pem" is in subdir "untrusted". Does the sort rely on something other than what is found in certdata.txt?

 

[Steen E]

Math mystery solved.

#!/bin/sh -e
_tb='    '
_nl='
'
LC_ALL=C
export LC_ALL
_lst=$(printf "\002\020\120\224\154\354\030\352\325\234\115\325\227\357\165\217\240\255" | vis -h)
_lst="${_lst#?}"
IFS="%"
for i in $_lst
do
    if test ${#i} -eq 2
    then _str="$_str $i"
    else
#convert ascii to hex - #1
# NB hexdump(1) reverse char sequence
        _str="$_str ${i%?} x"
        _hxstr="${i#??}$_hxstr"
    fi
done
_str="${_str# }"
#convert ascii to hex - #2
_hxstr=$(printf "$_hxstr"|hexdump -e '"%x"')
_lst=
IFS=" "$_tb$_nl
for i in $_str
do
    if test "$i" = "x"
    then
#convert ascii to hex - #3
    _tmp="${_hxstr#??}"
    _lst="$_lst ${_hxstr%$_tmp}"
    _hxstr="${_hxstr#??}"

    else _lst="$_lst $i"
    fi
done
echo "Serial for \"XRamp Global CA Root\" in nss format \"\002\020\120\224\154\354\030\352\325\234\115\325\227\357\165\217\240\255\""
echo
echo "valid: 50:94:6c:ec:18:ea:d5:9c:4d:d5:97:ef:75:8f:a0:ad"
echo "script convert:"
echo "$_lst"

First octet is a placeholder of some kind \002 and second octet seem to be number of hex ..places...

• \020 = 16 hex ..places.. (hextets?)

Ref: vis(1) https://forums.freebsd.org/help/bb-codes/#link-me

Actually don't know why /usr/src/security/caroot/MAca-bundle.pl choose to readin this octet series - the serial is checked against serials in the (arbitrary) bundle, but duplicates will only trigger a warning.

[edit] Ups, here's the right code:

printf "\002\020\120\224\154\354\030\352\325\234\115\325\227\357\165\217\240\255" \
    | hexdump -e '/1 "%02x" " "'

 

[nxjoseph]

I have a script that takes snapshots of ZFS dataset, marking it by current date&time and optionally with a comment. As I'm using /bin/tcsh for daily shell usage, I am trying to get used to it by using it in my scripts, for more, see my git repo; https://sr.ht/~yusufyaman/scripts

Example usage:

# takesnap
Pool to take snapshot from:
storage
Comment to add to the snapshot name:
backup after zfs copy
Will operate on storage
Date is: 2025-05-30-15:45
Comment is: backup-after-zfs-copy
Snapshot is taken: done!
storage@2025-05-30-15:45:backup-after-zfs-copy  0B      -       165G    -

# takesnap zroot
Comment to add to the snapshot name:

Will operate on zroot
Date is: 2025-05-30-15:41
Snapshot is taken: done!
zroot@2025-05-30-15:41  0B      -       96K     -

# takesnap zroot "test comment"
Will operate on zroot
Date is: 2025-05-30-15:41
Comment is: test-comment
Snapshot is taken: done!
zroot@2025-05-30-15:41:test-comment     0B      -       96K     -

#!/bin/csh

set date = `date +%F-%H:%M`

if ( "$#argv" > 0 ) then
    set pool = "$argv[1]"
else
    echo "Pool to take snapshot from: "
    set pool = "$<"
endif

if ( "$#argv" >= 2 ) then
    set comment = "$argv[2]"
    set isComment = 1
    set fixedComment = `echo "$comment" | sed 's/ /-/g'`
    set c = ":$fixedComment"
else
    echo "Comment to add to the snapshot name: "
    set comment = "$<"
    if ( "$comment" == "" ) then
        set isComment = 0
    else
        set isComment = 1
        set fixedComment = `echo "$comment" | sed 's/ /-/g'`
        set c = ":$fixedComment"
    endif
endif

if ( ! "$?isComment" ) then
    set isComment = 0
endif

if ( "$pool" == "" ) then
    echo "No pools given."
    exit 1
endif

echo "Will operate on $pool"
echo "Date is: $date"

if ( "$isComment" != 0 ) then
    echo "Comment is: $fixedComment"
    set takesnap = "zfs snapshot -r $pool@$date$c"
else
    set takesnap = "zfs snapshot -r $pool@$date"
endif
eval $takesnap
if ( "$status" != 0 ) then
    echo "Error taking snapshot of $pool"
    exit 1
else
    echo "Snapshot is taken: done!"
    if ( "$isComment" != 0 ) then
        set showsnap = "zfs list -H -t snap $pool@$date$c"
    else
        set showsnap = "zfs list -H -t snap $pool@$date"
    endif
    eval $showsnap
endif

exit 0

 

[Steen E]

A shell script to parse the nss bundle.

Mimicking the perl script /usr/src/secure/caroot/MAca-bundle.pl:

# s.parse_nss_bundle -f /tmp/certdata.txt -m
 Certificates in bundle: 176
 Certificates onhold: 41

A sort of logical evaluation:

# s.parse_nss_bundle -f /tmp/certdata.txt -l
 Certificates in bundle: 176
 Certificates onhold: 12

Serializing is much too slow:

# time s.parse_nss_bundle -f /tmp/certdata.txt -s
 Certificates in bundle: 176
 Certificates onhold: 41
        9.75 real         9.84 user         0.56 sys

# mkdir /tmp/store
# time /usr/src/secure/caroot/MAca-bundle.pl -i /tmp/certdata.txt -o store
##  Untrusted certificates omitted from this bundle: 41
##  Number of certificates: 135
        0.89 real         0.70 user         0.19 sys

s.parse_nss_bundle (ca 9kB):

#!/bin/sh -e
#   Interpreting escaped octal values relies on sh(1) builtin
#   'printf'. Transposing name of issuer to a kind of plain ascii is
#   implemented by setting LC_ALL and with help from sed(1).
#   Date format strings can be looked up at strftime(3)

#/usr/src/usr.sbin/certctl/certctl.sh:
# -> certctl.sh rehash
#/usr/src/secure/caroot/Makefile:
#|fetchcerts: .PHONY
#|    fetch --no-sslv3 --no-tlsv1 -o certdata.txt 'https://hg.mozilla.org/projects/nss/raw-file/tip/lib/ckfw/builtins/certdata.txt'

_script=s.parse_nss_bundle
_script_v="tst 01"
HELP="
A shell script to parse the nss bundle:

$_script -f <file> [-l|-m] [-d <dirname>] [-s|-a] [-v [<str>]]

Options:
 -l            - (logical) evaluate expiry plus trust tokens
 -m            - (mimick) evaluate trust token for 'server auth'
                 This is default
 -f <file>     - read from file
 -d <dirname>  - topdir to the two sort orders \"ok\" & \"onhold\".
                 Defaults to /tmp/nss_local
 -a            - (annotations) serialize both \"ok\" and \"onhold\"
                 certificates
                 Default is to only write \"onhold\" certificates
 -s            - serialize the bundle via calls to
                 \"openssl x509 -text -inform DER -fingerprint\"
                 Default is to only write the annotations
 -v <str>      - (version) use <str> to mark the certificates.
                 Leave empty to use timestamp of inputfile

The script sort certificates to one of two categories, \"ok\" & \"onhold\". The \"onhold\" certificates are annotated to document their lesser priority. Use 'head -n5 onhold/*|less' to see through the annotations.

BUGS
 Version string cannot begin with hyphen.
"
test "${0##*/}" = "$_script" ||
{
echo "Expecting script to have name:$_script. If you have sourced this file, then pls don't. Too many variables$HELP"
    return 1
}
# 3 causes for onhold + 1 not annotated
_ca_msg_nxplicit="# no explicit trust token"
_ca_msg_wdraw="# explicit trust token gainsaid:"
_ca_msg_expi="# Expired "

_ca_cnt=0
_ca_onhold_cnt=0

# nss octal encoded date format according to MAca-bundle.pl
#YYMMDDhhmmss plus a "Z"
_ca_sftm3="+%y%m%d%H%M%S"
_ca_now=$(date $_ca_sftm3)

# Date string written in cleartxt
# Tue Sep 01 12:00:00 1998
alias caroot_ctm='date -j -f "%a %b %d %H:%M:%S %Y"'

#/usr/src/secure/caroot/MAca-bundle.pl, the cli call
alias caroot_ossl='openssl x509 -text -inform DER -fingerprint'

# Pattern @[^[:alnum:]\-]@_@ also from MAca-bundle.pl
# work with sed(1) from locale C
alias caroot_sed="LC_ALL=C sed 's@[^[:alnum:]\-]@_@g'"

# use in pipeline - similar at MAca-bundle.pl
print_header_()
{
    cat - << EoStr
# idx: $_ca_idx ; mode: ${_logic:-mimick} - $_script version $_script_v${1:+${_nl}# CA: $@}
${_tjek#?}
EoStr
    cat -
}

_logic=
_if=
_all=
_seria=
_ca_dir=/tmp/nss_local
_ca_idx=
while test $# -gt 0
do
    case $1 in
    "-l") _logic=logical ;;
    "-m") : ;;
    "-f")
#        _if=$(readlink -f $2) ||
        _ca_idx=$(stat -f "%Sm" -t "${_ca_sftm3#?}" $2) ||
        {
echo "File not recognized, ..exiting$HELP"
            exit 1
        }
        _if=$2
        shift
    ;;
    "-d")
        _ca_dir=$2
        shift
    ;;
    "-s") _seria=1 ;;
    "-a") _all=1 ;;
    "-v")
        test "$2" = "${2#-}" &&
        {
            _ca_idx="$2"
            shift
        }
    ;;
    *)
echo "Args jumble, ..exiting$HELP"
        exit 1
    ;;
    esac
    shift
done
#return 0
test "$_if" ||
{
echo "Missing input file, ..exiting$HELP"
    exit 1
}
test "$_all" -a "$_seria" &&
{
echo "Args mixup - choose either of annotations or certificate formatting, ..exiting$HELP"
    exit 1
}

_nl='
'
_tb="    "

_ca_item_ok_dir=$_ca_dir/ok
_ca_item_onhold_dir=$_ca_dir/onhold

_ca_todo_label=
_ca_todo_msg=
_ca_todo_ln=
_ca_todo_calst=

IFS=
_s_flag=1
_e_flag=1
#_tjek
#_ln
#_label
# expiry
#_onhold
# conditionally set _s/e_onhold
#_dt_e
#_dt_s
# confounds 'distrust after' & explicit distrust
#_s_onhold
#_e_onhold

echo > /tmp/dbg.parsenss
while read -r tgt
do
    test "$tgt" &&
    {
        _tjek="${tgt#* }"
#{ condit - ln no blanks - begin
        if test "$_tjek" = "$tgt"
        then
            if test "$_ln"
            then
                if test "$tgt" = "END"
                then
                    _ca_todo_label="$_ca_todo_label$_nl${_label%?}"
                    test "$_seria" &&
                        _ca_todo_ln="$_ca_todo_ln$_nl${_ln#?}"
                    _ln=
                    _label=

                else _ln="$_ln$tgt"
                fi

            elif test "$_dt_s"
            then
                if test "$tgt" = "END"
                then
                    _dt_s=$(printf "$_dt_s")
                    test $_dt_s -lt $_ca_now &&
                        _s_onhold="# CKA_NSS_SERVER_DISTRUST_AFTER $_dt_s"
                    _dt_s=

# cut trailing Z
                else _dt_s=${tgt%?132}
                fi

            elif test "$_logic"
            then
                if test "$_dt_e"
                then
                    if test "$tgt" = "END"
                    then
                        _dt_e=$(printf "$_dt_e")
                        test $_dt_e -lt $_ca_now &&
                            _e_onhold="# CKA_NSS_EMAIL_DISTRUST_AFTER $_dt_e"
                        _dt_e=

# cut trailing Z
                    else _dt_e=${tgt%?132}
                    fi
                fi
            fi

# condit - ln no blanks
        else
            if test "${_tjek%% *}" = "Certificate"
            then _label="${_tjek#*\"}"

            elif test "$tgt" = "CKA_NSS_SERVER_DISTRUST_AFTER MULTILINE_OCTAL"
            then _dt_s=.
            elif test "$tgt" = "CKA_VALUE MULTILINE_OCTAL"
            then _ln=.

            elif test "${tgt%% *}" = "CKA_TRUST_SERVER_AUTH"
            then
                _tjek="${_tjek##* }"
                if test "$_tjek" = "CKT_NSS_TRUSTED_DELEGATOR"
                then _s_flag=

                elif test "$_tjek" = "CKT_NSS_NOT_TRUSTED"
                then _s_onhold="# $tgt"
                fi

#placeholder end-of-record
            elif test "${tgt%% *}" = "CKA_TRUST_STEP_UP_APPROVED"
            then
_ca_cnt=$(($_ca_cnt+1))
                if test "$_logic"
                then
# any non empty line will trigger onhold

#LN1 / LN2 / LN3 / LN4
#LN1
                    _ca_todo_msg="$_ca_todo_msg$_nl$_onhold"

#LN2 / LN3
                    if test "$_s_onhold" -a "$_e_onhold"
                    then
_ca_todo_msg="$_ca_todo_msg$_nl$_s_onhold$_nl$_e_onhold"

                    else
# TODO - ? too many condits - ? _s_onhold & _s_flag mutually exclusive
# evaluate explicit trust against 'prognostic' withdrawing that trust
# nota bene, when dateline is transgressed
# negates both evaluations
#                        if test ! "$_s_flag" -a "$_s_onhold"
                        if test "$_s_onhold" -a ! "$_s_flag"
                        then
_ca_todo_msg="$_ca_todo_msg${_nl}$_ca_msg_wdraw$_nl$_s_onhold"

                        elif test "$_e_onhold" -a ! "$_e_flag"
                        then
_ca_todo_msg="$_ca_todo_msg${_nl}$_ca_msg_wdraw$_nl$_e_onhold"

                        else _ca_todo_msg="$_ca_todo_msg$_nl$_nl"
                        fi
                    fi
#LN4
                    if test "$_s_flag" -a "$_e_flag"
                    then _ca_todo_msg="$_ca_todo_msg$_nl$_ca_msg_nxplicit"
                    else _ca_todo_msg="$_ca_todo_msg$_nl"
                    fi
                    _onhold=
                    _dt_s=
                    _s_onhold=
                    _dt_e=
                    _e_onhold=
                    _s_flag=1
                    _e_flag=1

                else
#TODO - begin

#LN1
                    if test "$_s_onhold"
                    then _ca_todo_msg="$_ca_todo_msg$_nl$_s_onhold"
                    else _ca_todo_msg="$_ca_todo_msg$_nl"
                    fi
#LN2
                    if test "$_s_flag"
                    then
_ca_todo_msg="$_ca_todo_msg${_nl}$_ca_msg_nxplicit"
                    else
_ca_todo_msg="$_ca_todo_msg${_nl}"
                    fi
                    _dt_s=
                    _s_onhold=
                    _s_flag=1

#TODO - fini
                fi

            elif test "$_logic"
            then
                if test "${_tjek#Not Valid After}" != "$_tjek"
                then
                    test $(caroot_ctm "${_tjek#*: }" "$_ca_sftm3") \
                        -lt $_ca_now && _onhold="$_ca_msg_expi${_tjek#*: }"

                elif test "$tgt" = "CKA_NSS_EMAIL_DISTRUST_AFTER MULTILINE_OCTAL"
                then _dt_e=.

                elif test "${tgt%% *}" = "CKA_TRUST_EMAIL_PROTECTION"
                then
                    _tjek="${_tjek##* }"
                    if test "$_tjek" = "CKT_NSS_TRUSTED_DELEGATOR"
                    then _e_flag=

                    elif test "$_tjek" = "CKT_NSS_NOT_TRUSTED"
                    then _e_onhold="# $tgt"

                    fi
                fi
            fi
        fi
#} condit - ln no blanks - end
    }
done < $_if

_ca_todo_ln="${_ca_todo_ln#$_nl}"
_ca_todo_msg="${_ca_todo_msg#$_nl}"
_ca_todo_calst="${_ca_todo_label#$_nl}"
_ca_todo_label=$(printf "${_ca_todo_label#$_nl}"\
    | caroot_sed)

#tee << EoStr >> /tmp/dbg.parsenss
#LN
#$_ca_todo_ln
#MSG
#$_ca_todo_msg
#LABEL
#$_ca_todo_label
#EoStr
#exit

test -d "$_ca_item_ok_dir" || mkdir -p "$_ca_item_ok_dir"
test -d "$_ca_item_onhold_dir" || mkdir -p "$_ca_item_onhold_dir"

if test "$_seria"
then
#_flag
#_tjek
#_res
IFS=" ""$_tb"$_nl
    for i in $_ca_todo_label
    do
        _tjek=
        _res=$_ca_item_ok_dir
        for ii in 1 2${_logic:+ 3 4}
        do
            _flag="${_ca_todo_msg%%$_nl*}"
            test "$_flag" && _res=$_ca_item_onhold_dir
            _tjek="$_tjek$_nl${_flag:-#}"
            _ca_todo_msg="${_ca_todo_msg#*$_nl}"
        done
        test "$_res" = "$_ca_item_ok_dir" ||
            _ca_onhold_cnt=$(($_ca_onhold_cnt+1))
#{
        printf "${_ca_todo_ln%%$_nl*}" \
        | caroot_ossl \
        | print_header_ \
        > $_res/$i.pem
#} &
        _ca_todo_ln="${_ca_todo_ln#*$_nl}"
    done

else
#_flag
#_tjek
#_res
IFS=" ""$_tb"$_nl
    for i in $_ca_todo_label
    do
        _tjek=
        if test "$_all"
        then _res=$_ca_item_ok_dir
        else _res=
        fi
        for ii in 1 2${_logic:+ 3 4}
        do
            _flag="${_ca_todo_msg%%$_nl*}"
            test "$_flag" && _res=$_ca_item_onhold_dir
            _tjek="$_tjek$_nl${_flag:-#}"
            _ca_todo_msg="${_ca_todo_msg#*$_nl}"
        done
        test "$_res" != "${_all:+$_ca_item_ok_dir}" &&
            _ca_onhold_cnt=$(($_ca_onhold_cnt+1))
        test "$_res" &&
        {
            echo | print_header_ "${_ca_todo_calst%%$_nl*}"\
            > $_res/$i.pem
        }
        _ca_todo_ln="${_ca_todo_ln#*$_nl}"
        _ca_todo_calst="${_ca_todo_calst#*$_nl}"
    done
fi
printf " Certificates in bundle: $_ca_cnt
 Certificates onhold: $_ca_onhold_cnt
"

Spoiler: Notes

When reading the README file at nss-3.104/nss/lib/ckfw/builtins the puzzle bits finally revealed some sketchy picture. Before that I had, at the start of my worknotes,

# Expiry:
# Seemingly harmless, not technical, just an arbitrary rulesetting
# Distrust:
# Inexplicable worthing&context, especially naming a date after which
# distrust occur - is it technical? is it a blacklist notion?

This is not going to be very technical, and it has nothing to do with cryptography, and there is much complexity in the 'secure socket layer' even without cryptography. The complexity should be described in form of technical details, where the following story can only offer references to further reading.
(See for example rfs9618 for an illustrated! example of complex logistics)
https://www.rfc-editor.org/rfc/rfc9618

The meaning of words is always debatable.
In the scripted - and non-cryptographic - context of evaluating a certificate, there are two keywords, expiry and trust, and the readme-file at the Mozilla project "Network Security Services" subdir could clarify some of the uncertainty. In short, it says that the literal text trust tokens - that is, the entries in certdata.txt - are settable by the Mozilla project or anyone else, providing they have the time and expertise for doing so.

This bit of information "from the horse's mouth" correlate well with my impression of the ssl scenario:
Certificates are third-party agents in a principally two-sided communication scheme - although the digital reality, from data layer to application layer, is an evasively disrupted landscape of cris-crossing communications. They concern a technical feature, cryptography, and date of expiry could be conjectured to relate to some algorithmic assumption of potential 'cryptograhic' compromise. The nss tokens of distrust seem to introduce a fourth-party evaluation - neither of CA, enduser or homepage owner.

.. though the idea about expiry being somehow related to cryptography is a sillyness.. the formal date of expiration is just an announcement from the CA that their service is available untill that date.

There is another bit of information to note, from the same source, namely the conjecture or prognostic that there will be some kind of delay - even on the scale of years - before an expiry is finally settled. The meaning is not very clear, but what does seem clear, is, how silly it may sound, the certainty of uncertainty.

The hard facts about delays in expiry is exemplified in the case of one of the root certificates used by Let's Encrypt.
https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/
The notion of complexity in logistics is also profiled in the short introductory page at 360 Total Security.
https://caprogram.360.cn/#plan

The scripted and non-cryptographic installation of certificates in the base system reflect this uncertainty by explicitly installing 'untrusted' certificates, ie. they are not only part of the git repository, in subdir /usr/src/secure/caroot/untrusted, but is also part of base system after a 'make world'. (As long as your src.conf does not have WITHOUT_CAROOT) It is sort of an simple uncertainty, concretely installing 'untrusted' certificates to the base system.

There are specialized programmatic routines for handling communication via ssl, and I think it is most typical that cryptographic evaluation and logistic evaluation is mixed together. The logistic part may be referred to as "certificate path building", according to
openssl-verification-options(1) or just "chains of trust". https://letsencrypt.org/certificates/
That is the more complex uncertainty.

An exception to this assumed rule is the perl script MAca-bundle.pl that evaluate trust tokens without processing the cryptographic elements - although I must note that I do not fully understand the perl script. It seems to be a simple file parsing routine. Mimicking the perl script ran into trouble. I discovered the case of "Baltimore CyberTrust Root" vs "Certum Root CA" both flagged in certdata as trusted in relation to 'email protection' and only 'must verify' trust in relation to 'server auth' and both certificates with the 'distrust after' tokens set to false, ie. for no specific date. The certificate from Certum was deselected by the perl script whereas the one from Baltimore was kept and the trivial explanation was that the nss bundle had been updated. There is no documentation or versioning of the base system caroot - it is just the FreeBSD 14.2-p3 srcversion - but I checked the certdata.txt from ports security/ca_root_nss, which use "nss-3.104", where the Baltimore certificate did have a trust token set for 'server auth'.

While the perl script deselect, the actual installation to the base system merely deprioritize. All certificates from the nss bundle are installed.
Base system deprioritize:

• "Certum Root CA", no 'distrust after', with trust token 'email protection'
• "D-TRUST Root CA 3 2013", no 'distrust after', with trust token 'email protection'
• "Staat der Nederlanden Root CA - G3", no 'distrust after', with trust token 'email protection'

The deselection / deprioritizing accord with the perl script conditional, summarized as ($maytrust and not $distrust) since the script does not parse for the trust token relating to 'email protection'.

The perl script also do not parse for an expired certificate. There is however only two expired certificates. One of them is the Baltimore certificate - expired May 12 - and the other an awkward "Explicitly Distrust DigiNotar Root CA", expired March 31.

 

[gotnull]

Hi,

I like getting information about ports through pkg-info(8), unfortunately this command only works for installed ports, for uninstalled ports I don't know any other option than going online to freshports website.
This script use pkg-rquery(8) to mimics regular pkg-info(8) except gathering things that I don't need(like shared libraries, annotations, required-by, and options), the rest looks almost like pkg-info(8), basically the script calls pkg-info(8) when the port is already installed on the system or calls a internal function if the package is not present on the system.
So now in any case I can get the ports's info I am looking for, to me what's important is mainly the description, the URL of the project, the version, the dependencies (feel free to modify it if you need more detailed info).
It's not perfect, you'll probably see few quirks sometimes, but for my basic use case it does the job, and I hope it will for you too.

See below an example, ppo is an alias for pkg_rinfo:

~ > ppo uget
Name          :uget
Version       :2.2.3_2
Origin        :net/uget
Architecture  :FreeBSD:14:amd64
Prefix        :/usr/local
Licenses      :LGPL21+
Maintainer    :ports@FreeBSD.org
WWW           :https://ugetdm.com/
Size          :1.33MiB

Description: uGet is a very powerful download manager application, with large inventory
of features but is still very light-weight and low on resources.

General features:
- Downloads queue
- Resume downloads
- Advanced download categories
- Clipboard monitor
- Batch downloads
- Multi-protocol
- Scheduler

Dependencies:
x11-toolkits/pango
x11-toolkits/gtk30
www/aria2
print/harfbuzz
graphics/gdk-pixbuf2
graphics/cairo
ftp/curl
devel/libnotify
devel/glib20
devel/gettext-runtime
accessibility/at-spi2-core

Here is the script:

#!/usr/bin/env sh
# NAME: pkg_rinfo aka pkg remote info
# DESCRIPTION: a pkg-info like working with both installed or uninstalled ports
# DEPENDENCIES: none
# AUTHOR: gotnull
# LICENSE: Unlicense

set -eu
# set -v
# set -x

usage_() {
cat << EOF
Usage: ${0##*/} package_name

Example:
        # get info on port called perlconsole
        ${0##*/} perlconsole

        # works also with origin name
        # sometimes it gives better result
        ${0##*/} devel/perlconsole
EOF
exit 1
}

pkg_rinfo_() (
        # check if the port has dependencies
        if [ "$(pkg rquery '%?d' "$1")" -eq 1 ] ; then
                print_dep=on
        else    print_dep=off
        fi

        # gather some useful information
        p1=$(pkg rquery 'Name :%n\nVersion :%v\nOrigin :%o\nArchitecture :%q\nPrefix :%p\nLicenses :%L\nMaintainer :%m\nWWW :%w\nSize :%sh\n' "$1" | column -t)
        p1_1=$(pkg rquery 'Description: %e' "$1")
        p2=$(printf "%s\n" "Dependencies: ")
        p2_2=$(pkg rquery '%do' "$1")


        # display information
        case $print_dep in
                on)  printf "%s\n\n%s\n\n%s\n%s\n" "$p1" "$p1_1" "$p2" "$p2_2" ;;
                off) printf "%s\n\n%s\n" "$p1" "$p1_1" ;;
        esac
)

check_port() (
        # determine if port is installed
        if pkg info | grep -q "$1" ; then
                which_pkg_info=system
        else    which_pkg_info=custom
        fi

        # pick the regular pkg-info or not depending on the previous test
        case "$which_pkg_info" in
                system) pkg info "$1" ;;
                custom) pkg_rinfo_ "$1" ;;
        esac
)

# be sure that an argument is provided
if [ "$#" -ne 1 ] ; then
        usage_
fi

# checking if the port even exists
if ! pkg rquery %n "$1" > /dev/null 2>&1 ; then
        printf "%s\n" "Error: $1 isn't a valid port."
        exit 1
fi

check_port "$@"

When pkg_rinfo invoked, how to know if the answer is coming from pkg-info or pkg_rinfo?
1) pkg-info mentions that the port is already installed with a line starting with "Installed on ...", pkg_rinfo doesn't.
2) pkg-info is more verbose than pkg_rinfo because of the shared libraries, required, annotations and options.

 

[Rudy]

I posted jmem - Memory usage for jails in another thread... Nice CLI interface so show off your jails and see what versions need some updating.

 

[fygar256]

A wrapper for yt-dlp on GUI DE.

#!/usr/bin/env python3
import subprocess

savedir="/home/gar/Videos"
print("Youtube Downloader")
url=input("Input URL:")
subprocess.call(["/usr/local/bin/yt-dlp","-P",savedir,"--trim-filenames","32",url])
print("Done. Hit enter key")
a=input()

 

[gotnull]

Before everything you should note that I took inspiration from an article written by Dru Lavigne, so if you want more details about how this works feel free to read it.
I wrote this script because I wanted something easier and quicker than typing everything by hand every time.
The goal here is to upgrade the packages via pkg safely, without been worried about breaking things, because as you may already know from time to time updates can be surprising.

So the main idea is to run pkg upgrade in a dedicated boot environment(yes ZFS is required), reboot into it temporarily and check how it goes.
Now at this point there are two solutions:
_ everything is fine then you can turn the temporary boot environment into permanent by issuing the command bectl activate boot_environment_name.
_ the update went wrong, something is broken, in this case just reboot and you'll be back into your previous working boot environment like nothing happened. After that it's up to you, either you keep the faulty boot environment to investigate, or you get rid of it completely by running the command bectl destroy boot_environment_name.

I hope I explained this well, most actions are commented in the script so it should be easy to follow.

#!/usr/bin/env sh
# NAME: pkg_safe_upgrade
# DESCRIPTION: upgrade your packages safely via boot environment
# Credit goes to Dru Lavigne which I took inspiration from in her article:
# https://klarasystems.com/articles/managing-boot-environments
# DEPENDENCIES: none
# AUTHOR: gotnull
# LICENSE: Unlicense

set -eu
#set -v
#set -x

# handle error messages
error() {
    printf "Error: %s\n" "${1:-"unknown error"}"
}

# root privileges are required
check_root() {
    [ "$(id -u)" -eq 0 ] || { error "root privileges needed."; exit 1; }
}

# prepare the boot environment
setup_be() {
    # pretty name for our BE
    be_name=pkg_upgrade_$(date +%d-%m-%Y_%H%M%S)

    # create a new BE
    bectl create "$be_name"

    # get the mount point
    mount_point=$(bectl mount "$be_name" | awk '{print $NF}')
}

# main function
main() {

    # pkg update in BE
    pkg -r "$mount_point" update -f || { error "cannot update"; return 1; }

    # determine if there is actually any package to upgrade
    pkg_nb=$(pkg -r "$mount_point" upgrade -n | grep 'Number of packages to be upgraded' | awk '{print $NF}')

    # if yes, proceed
    if [ -n "$pkg_nb" ] && [ "$pkg_nb" -ne 0 ] ; then
        # pkg upgrade in BE
        pkg -r "$mount_point" upgrade || { error "cannot upgrade"; return 1; }

        # umount BE
        bectl umount "$be_name" ||  { error "cannot umount $be_name"; return 1; }
        if [ -d "$mount_point" ] ; then
            rmdir "$mount_point" || { error "cannot delete directory in /tmp $mount_point"; return 1; }
        fi
        # activate temporary only the new BE
        bectl activate -t "$be_name" || { error "cannot activate $be_name"; return 1; }
        echo "pkg upgrade: done."
        exit 0

    # if no, umount and delete the new BE
    else
        bectl umount "$be_name"  || { error "cannot umount $be_name"; return 1; }
        bectl destroy "$be_name" || { error "cannot destroy $be_name"; return 1; }
        [ -d "$mount_point" ] && rmdir "$mount_point"
        echo "pkg upgrade: no updates needed."
        exit 0
    fi
}

# a bit of cleaning, use the force flag to get rid of everything about the boot environment
cleanup() {
    bectl umount -f "$be_name"
    bectl destroy -F "$be_name"
    [ -d "$mount_point" ] && rm -rf "$mount_point"
    echo "cleaning: done."
    exit 0
}

# only root is allowed to run this script
check_root

# destroy BE if it can't be set properly
setup_be || { error; cleanup; }

# cleaning measure in case an interruption happened
trap 'cleanup' INT

# run the main function, but if any error occurred during the process remove everything
main || cleanup

EDIT: issues fixed.

 

[gotnull]

The main goal of this script quick_log is to have an easy access to your logs in one single command.
I use it often when "debugging", because it does not require a lot of typing to get what I am searching for, straight to point.
For this to work, a fuzzy finder tool textproc/fuzzy is used.

When the script starts it lists all the logs on the systems waiting for you to make a choice, once you selected one (either by typing its name or moving the cursor UP/DOWN with arrows keys or CTRL+p/n), it opens it up.

Whether they archived(.bz2) or not(.log) logs can be read.
PF logs /var/log/pflog are also readable.
For more convenience logs are displayed in reverse order, so recent events come first.
To quit when reading log press q, or if you are in the list just press ESC.

This is how it looks:

At start it lists logs present on the machine

user@fbsd:~ $ qlog
log:
/var/log/utx.log
/var/log/utx.lastlogin
/var/log/cron
/var/log/lpd-errs
/var/log/security
/var/log/maillog
/var/log/pflog.0.bz2
/var/log/userlog
/var/log/auth.log
/var/log/xferlog

Typing the letter "m" filters the list

user@fbsd:~ $ qlog
log: m
/var/log/maillog
/var/log/messages
/var/log/maillog.0.bz2
/var/log/maillog.1.bz2
/var/log/daemon.log

Here pflog has been selected

      1 18:59:50.566735 rule 0/0(match): block in on vtnet0: xxx.xxx.xxx.xxx.51718 > 224.0.0.251.5353: 7267 [|domain]
      2 18:59:50.566002 rule 0/0(match): block in on vtnet0: xxx.xxx.xxx.xxx.52228 > 224.0.0.251.5353: 7266 [|domain]
      3 18:59:50.528895 rule 0/0(match): block in on vtnet0: xxx.xxx.xxx.xxx.40565 > 224.0.0.251.5353: 7265 [|domain]
      4 18:59:50.486355 rule 0/0(match): block in on vtnet0: xxx.xxx.xxx.xxx.38979 > 224.0.0.251.5353: 7264 [|domain]
      5 18:59:50.400437 rule 0/0(match): block in on vtnet0: xxx.xxx.xxx.xxx.45704 > 224.0.0.251.5353: 7263 [|domain]
      6 18:59:50.357500 rule 0/0(match): block in on vtnet0: xxx.xxx.xxx.xxx.42738 > 224.0.0.251.5353: 7262 [|domain]
      7 18:59:50.268274 rule 0/0(match): block in on vtnet0: xxx.xxx.xxx.xxx.37805 > 224.0.0.251.5353: 7261 [|domain]
      8 18:59:50.225376 rule 0/0(match): block in on vtnet0: xxx.xxx.xxx.xxx.60248 > 224.0.0.251.5353: 7260 [|domain]
      9 18:59:50.139628 rule 0/0(match): block in on vtnet0: xxx.xxx.xxx.xxx.45631 > 224.0.0.251.5353: 7259 [|domain]
      ...

That's it, I think you get the idea.

Here is the script:

#!/usr/bin/env sh
# NAME: quick_log aka qlog
# DESCRIPTION: Quick access to the logs without bothering with names, path or extensions,
# it can also read pf logs.
# DEPENDENCIES: fzy, doas(optional)
# AUTHOR: gotnull
# LICENSE: Unlicense

set -eu
# set -x
# set -v

# error messages
error() {
    printf "Error: %s\n" "${1:-"unknown error"}"
}

# verifying dependencies
check_dependencies() {

    # running the script as a normal user requires doas
    if [ "$(id -u)" -ne 0 ] ; then
        if ! command -v doas >/dev/null ; then
        error "missing depency security/doas or security/opendoas"
        return 1
        fi
    fi

    # fzy is mandatory
    if ! command -v fzy >/dev/null ; then
        error "missing depency textproc/fzy"
        return 1
    fi
}

# determine if doas is used
set_cmd() {

    if [ "$(id -u)" -eq 0 ] ; then
        cmd=""
    else cmd="doas"
    fi
}

# manage pf logs
read_pflog() {

    # archived pf logs
    if [ "${logfile##*.}" = bz2 ] ; then
        $cmd zcat "$logfile" | $cmd tcpdump -ne -r - | tail -r | less -N
        exit 0

    # not archived pf logs
    else $cmd tcpdump -ne -r "$logfile" | tail -r | less -N
        exit 0
    fi
}

# manage normal logs
read_log() {

    # archived logs
    if [ "${logfile##*.}" = bz2 ] ; then
        $cmd zcat "$logfile" | tail -r | less -N
        exit 0

    # not archived logs
    else $cmd tail -r "$logfile" | less -N
         exit 0
    fi
}

# main function
main() {

    set -- "$($cmd find /var/log/ -type f)"
    logfile=$(printf "%s\n" "$@" | fzy --prompt="log: " --lines=10)


    if [ -n "$logfile" ] ; then

        if echo "$logfile" | grep -q pflog ; then
            read_pflog
        elif echo "$logfile" | grep -q utx ; then
            error "cannot read ${logfile##*/}."
            return 1
        else read_log
        fi
    else exit 0
    fi
}

check_dependencies || exit
set_cmd
main || exit

I hope it serve you well.
Have fun.