Useful scripts

Hi people!
Please, post here useful scripts which can help to do some useful or difficult operation on FreeBSD OS.
Found a script on one of my old laptops to attach/insert a geom(4) I/O scheduler - RTFM gsched(8).
  • An I/O scheduler is at least useful on a UFS (or alike) system; I don't know and didn't try with ZFS. Other use cases might be a DB on a dedicated geom(4) device.
  • The I/O scheduler can significantly increase throughput and interactivity with concurrent tasks, as described by the authors of gsched(8).
    I can could verify that - the system runs much "smoother" when running background tasks with heavy disk I/O. The script served me well for years.
  • The script might violate some guidelines for rc scripting that I was not aware of.
  • Very Likely it has bugs, since I was the only user.
    Drop me a note and I'll fix that.
  • If a few others could prove it's safe, it could go to /usr/share/examples/etc/rc.d or even in the base.
  • Suggested directories:{/usr/local}/etc/rc.d/geom_sched, {/usr/local}/libexec/fs_summarize.awk
    or the respective directories in /usr/share/examples
    and rc.geom_sched.conf is for insertion in rc.conf{.local}
The other script fs_summarize.awk is useful to determine the parameters for newfs(8) or just if you want some info about a FS or directory.
Just in case this is needed: the standard FreeBSD BSD license applies to the scripts, the snippet rc_geom_sched.conf obviously does not need any license.
 

Attachments

Moving UNIX domain sockets
Not really a script as such but more an example that UNIX domain sockets can be moved (not copied). This is really handy for allowing X11 access into a Jail.

Code:
UNIQUE=$$
Xephyr :$UNIQUE &

# Wait until the socket exists

mv /tmp/.X11-unix/X$UNIQUE /jails/myjail/tmp/.X11-unix/X0
jexec myjail xterm -display :0

Finding absolute path of script or program
Also, a handy way to get the absolute path to your script (or more usefully, the prefix) without faffing with procfs and all that other stuff. This is useful because sometimes relative paths are not supported or not desired.

Code:
PREFIX="$(cd "$(dirname "$(which "$0")")" && cd .. && pwd)"

# which "$0" ensures that a path is returned if the program was run from $PATH
# dirname ensures that the directory containing the program (i.e bin) is returned
# cd, cd .., pwd ensures that an absolute path is given by going into the directory and using pwd.

It seems a bit "dirty" but I also use this (via popen) in C++ and perl. It is the only guaranteed way that I have found over the years. And I have tried quite a few!
 
Not really a script as such but more an example that UNIX domain sockets can be moved (not copied). This is really handy for allowing X11 access into a Jail.

Code:
[...]
mv /tmp/.X11-unix/X$UNIQUE /jails/myjail/tmp/.X11-unix/X0
Does that also work if /tmp and /jails are on different file systems? Because in that case a you’re technically copying, not moving.
Also, a handy way to get the absolute path to your script (or more usefully, the prefix) without faffing with procfs and all that other stuff. This is useful because sometimes relative paths are not supported or not desired.

Code:
PREFIX="$(cd "$(dirname "$(which "$0")")" && cd .. && pwd)"
The realpath(3) function (and the corresponding realpath(1) command) is very useful in this context. It converts relative paths to absolute paths and resolves symbolic links. In shell scripts I'm using the following to find out the script’s directory:
Code:
SCRIPTDIR=$(realpath "${0%/*}")
So, to get the prefix, that would be rather simple:
Code:
PREFIX=$(realpath "${0%/*}/..")
Sometimes I resort to writing a script in zsh, because FreeBSD's sh is missing quite a few features (it’s not a POSIX-compatible shell). In that case, the prefix can be found without calling any external programs, so this is most efficient:
Code:
PREFIX=${0:A:h:h}
Explanation: Basically, the “:A” modifier is equivalent to realpath(3), and the “:h” modifier is equivalent to dirname(3).

By the way, in Python you would do this:
Code:
from sys import argv
from os.path import realpath, dirname

prefix = dirname(dirname(realpath(argv[0])))
Other languages like Perl or Ruby also have bindings to the standard library functions realpath(3) and dirname(3), so it basically works the same there, too (modulo different syntax, of course). C and C++ can use those functions directly anyway, of course – the former is in <stdlib.h> and the latter is in <libgen.h>. Both are POSIX / SUS standards.
 
Does that also work if /tmp and /jails are on different file systems? Because in that case a you’re technically copying, not moving.
Unfortunately no, I don't think that is possible. Perhaps with something like nullfs? You can trick it into opening the socket on the other FS?

Perhaps it is time for Zirias's project to come in handy? :): https://forums.freebsd.org/threads/new-tool-remusock-remote-unix-socket-access.76026/


The realpath(3) function (and the corresponding realpath(1) command) is very useful in this context. It converts relative paths to absolute paths and resolves symbolic links.

From what I could see, this doesn't work for things in PATH? I.e realpath ls returns /home/kpedersen/ls
 
From what I could see, this doesn't work for things in PATH? I.e realpath ls returns /home/kpedersen/ls
Yes, it does work for things in PATH. When you call a shell script via PATH, $0 always includes the directory in which it was found.
 
By the way, some generic advice for writing shell scripts, for those wo are not already experts … ;)

Try to write shell scripts using /bin/sh only. This will make the scripts most portable and doesn't require people to install a specific shell if you share them with others. Note that /bin/sh scripts will usually work without change with POSIX-compatible shells, too (zsh, ksh, bash). If you absolutely have to use POSIX features that are not supported by FreeBSD's /bin/sh, I suggest using ksh because it most closely resembles the POSIX standard, and also works with bash and zsh. Avoid using bashisms or zshisms that don't work anywhere else.

Begin your script with set -Cefu.
  • -C (“noclobber”) prevents the script from overwriting files with > accidentally. If you need to overwrite a file on purpose and you're absolutely sure that it’s the correct thing to do, you can use >| instead to override the -C option, or enclose the respective code between set +C and set -C, or simply rm -f the file beforehand.
  • -e (“errexit”) causes the script to terminate immediately if a command returns an error (i.e. a non-zero return code). See the sh(1) manual page for details. If you know that a command may return a non-zero exit code, but you want to continue anyway, append || true, or (better) put it into an if clause and print a message to the user if appropriate.
  • -f (“noglob”) disables globbing, also known as pathname expansion or filename generation using wildcards like *. This is often a cause of subtle bugs and malfunctions, so it’s best to disable it altogether. To create a list of files, it’s better to use things like ls | grep ... or find(1) which is much more powerful.
  • -u (“nounset”) causes the script to print an error message and exit immediately if you try to use a variable that has not been set before. unset FOO; echo $FOO will trigger this, for example. This is useful to quickly detect typos in variable names. If you need to expand a variable that might be unset (and you known what you’re doing), ${FOO-} will prevent the error (if FOO is unset, it will expand to an empty string). Again, see the sh(1) manual page for details.
These options are very helpful to write scripts that are more robust and have a lower risk of malfunctions.

The next line in most of my scripts is: ME=${0##*/}
That sets the variable ME to the basename of the script, so it can conveniently be used in error messages and similar, like in this little helper function:
Code:
Err ()
{
        echo "${ME}: $*" >&2
        exit 1
}
So you can write things like this:
Code:
test -f "$INFILE" || Err "File not found: $INFILE"

If in doubt, always write variable expansions in double quotes.
If a variable might begin with a dash, use -- to prevent it from being confused with an option if appropriate, e.g.:
ls -l -- "$MYDIR"

By the way, do not use backticks ( `foo`) for command substitution. Nesting them is difficult, they can be confused with single quote characters ( 'foo' looks very similar), and the quoting rules involving backticks are complicated. Better use the alternative $(foo) syntax. This can be nested arbitrarily deep without having to use heaps of backslashes, and the quoting rules are more intuitive.

If you need to debug a shell script, the -v and -x options are very useful: sh -vx ./myscript
  • -v causes the script code to be printed as it is read from the file during execution.
  • -x causes the resulting commands (after variable expansion etc.) to be printed right before they're executed. This is especially useful to see what your script is actually doing.

Happy hacking!
 
Sleep daemon for monitoring battery life and putting laptop to sleep when needed

Here's a useful set of scripts for creating a daemon to monitor battery life and sleep the system when it's critically low that works without a desktop (it also works with TWM, but it's not required). The original discussion on this topic is here, the workhorse is the monitor, which is a modified version of the script DutchDaemon originally posted here. The script basically checks the battery level every 5 minutes and when it drops to 10%, it sets a 2 minute timer and checks again, if it's getting lower at that time, it sends an email to root warning of imminent shutdown and 2 minutes after that sleeps the system.

I use the daemon because I have not been using a DE lately and my laptop kept powering down when the battery got low. This keeps that from happening.

The zipfile - sleepd.zip contains these three files:

1. rc-sleepd - the rc script to control the daemon, gets installed in /etc/rc.d
2. sbin-sleepd - the script that does the actual monitoring, gets installed in /usr/sbin
3. install.sh - a simple install script to install the daemon and remind you how to enable the service

To install, just download the zipfile as a normal user with wheel privileges and:

unzip sleepd.zip
cd sleepd
sudo sh ./install.sh


and

sudo sysrc -f /etc/rc.conf sleepd_enable="YES"
sudo service sleepd start


20200722-1418 wds - updated the scripts to use /usr/local prefixes in line with hier()
 

Attachments

OpenSSH on Windows is kinda bearable compared to remote powershell. The biggest issue is no native tmux / screen support.
I was not happy with the existing practice of using a tabbed console or tmux on the local host and opening multiple ssh connections so I made my own tool.


Basically run bin/vimux and then you can use tmux-like shortcuts like ctrl-c to create new terminals within the same session.

(the underlying platform enabling this is vim and its virtual-pty support ;)
 
In another thread I mentioned this shell function, and mjollnir asked me to put it here, so here it is.
It uses zsh syntax and is meant to be put in your ~/.zshrc, but I guess it could be adapted to bash without much effort.

Alternatively you can turn it into a standalone script: Remove the first line “inplace ()” and the outermost level of braces, and prepend the line “#!/usr/bin/env zsh”. Save the script with the name inplace in some bin directory (so it can be found via your $PATH setting), and make it executable: chmod 755 inplace. Make sure you have shells/zsh installed.
Code:
inplace ()
{
        #   Execute a command that reads from a file and writes to stdout,
        #   and move the output back to the input file.  The file name must
        #   be the last argument.  Timestamps (atime, mtime) and permissions
        #   are preserved.  A backup of the original file is retained with the
        #   extension ".BAK".  If an error occurs, the orignal file is not
        #   changed, and a temporary file is kept in the same directory
        #   (unless it is empty).

        emulate -L zsh
        local LASTARG="${@[-1]}"
        local BAKNAME="$LASTARG".BAK
        local TMPNAME="$LASTARG".inplace_tmp

        echo "Note: Keeping backup in $BAKNAME" >&2
        rm -f -- "$TMPNAME" \
        && "$@" > "$TMPNAME" \
        && touch -r "$LASTARG" -- "$TMPNAME" \
        && chmod 0$(stat -f"%OLp" -- "$LASTARG") "$TMPNAME" \
        && mv -f -- "$LASTARG" "$BAKNAME" \
        && mv -f -- "$TMPNAME" "$LASTARG" \
        || {
                if [[ ! -s "$TMPNAME" ]]; then
                        rm -f -- "$TMPNAME"
                fi
                return 1
        }
}
Usage example:
inplace iconv -f ISO8859-16 -t UTF-8 somefile.txt
 
Here's my current backup script in case it helps other new users with similar needs.

All of my data from multiple computers fits on a single hard drive, so I back it all up to a central FreeBSD server with a removable drive tray and store the drives offsite.

The morning protocol takes about 15 seconds; if the programmable LED is flashing green, I open the tray and swap the drive, then take it with me when I leave. At night I bring home an older backup drive and then repeat the swap procedure the next morning.

The script runs zfs scrub on insertion every Saturday, so all of the backup drives get scrubbed for errors once every few weeks as they rotate through.

I'm currently backing up a Windows desktop, an Ubuntu laptop, a FreeBSD server, and the backup server itself to the external drive.

The system detects drive insertion using this devd config:

Bash:
# cat /usr/local/etc/devd/backup.conf
notify 100 {
    # insert backup drive
    match "system" "GEOM";
    match "subsystem" "DEV";
    match "type" "CREATE";
    match "cdev" "gpt/backup";
    action "su -l backup -c 'doas /usr/local/sbin/backup.sh --on-insert' &";
};

notify 100 {
    # eject backup drive
    match "system" "GEOM";
    match "subsystem" "DEV";
    match "type" "DESTROY";
    match "cdev" "gpt/backup";
    action "su -l backup -c 'doas /usr/local/sbin/backup.sh --on-eject' &";
};

The script runs using doas for a dedicated user with this config:

Bash:
# cat /usr/local/etc/doas.conf
permit nopass backup as root cmd /usr/local/sbin/backup.sh

The backup script itself is attached below.
 

Attachments

Last edited:
Here is a script to run in the terminal to give you some info on what your named server is doing.

Perl:
#!/usr/local/bin/perl
#
# Quick script to see that the DNS server is doing
# Fri Oct 30 16:34:29 PDT 2020, created, rudy
#
# requirements:
# pkg install p5-Term-ReadKey
# pkg install p5-Term-ANCIScreen
# ----------------------------------------------------------------------

use Term::ReadKey;
use Socket;
use POSIX qw(ceil floor);
use Term::ANSIScreen qw/:color :cursor :screen :keyboard :constants/;
use strict;

our (%h, %r, %lh, %lr);   # hits, requests, and last for those two
our (%dns, %stats);       # DNS cache, and RDNC output
our ($width, $height);   # terminal size
our $hostname = `hostname`;
chomp($hostname);

# set 'network' to your DNS server's ip block
# example, your DNS server is 10.8.2.3 or 10.8.2.4 then use
our $network = '10.8.1.0/28';  # DNS server Range (used in TCPDUMP)
our $port = 53;                   # not sure this will ever change
our $countPerCycle = 1000;        # count flag for TCPDUMP

# ----------------------------------------------------------------------

print "Welcome, starting TCPdump to gather snapshot of current DNS requests.\nStandby ......\n\n";
while (1) {
    &tcpdump;
    my ($lastwidth, $lastheight) = ($width, $height);
    ($width, $height) = GetTerminalSize ();
    print cls() if ($height != $lastheight || $width != $lastwidth);  #wipe screen on resize
    locate 1, 1;  # move cursor to top left
    &getStats;
    &header;
    &top10;
    &footer;
}

sub tcpdump {
    %lr = %r;
    %lh = %h;
    open TCPDUMP, "tcpdump -n -c $countPerCycle dst port $port and dst net $network 2> /dev/null |" or die "No tcpdump\n";
    while (<TCPDUMP>) {
        # 13:14:18.664904 IP 52.119.118.12.42467 > 208.69.40.3.53: 26522+ A? l3.dca0.com. (29)
        # this could probably use refinements... version 0.1
        my @a = split(' ');
        my $ip = $a[2];
        $ip =~ s/(\.\d+)$//; #remote port;
        $h{$ip}++; # easy to spoof...
        if (/Flags/) {
            $r{'Flags'}++;
        } elsif ($a[6] eq '[1au]') {
            $r{"$a[7] $a[8]"}++;
        } else {
            $r{"$a[6] $a[7]"}++;
        }
    }
    close TCPDUMP;
}


sub top10 {
    # started out as top10, but let's just max out to match terminal size.
    my $top =  floor($height/2) - 3;

    # print top IPs doing requests
    print colored ['black on white'], sprintf("%5s      %17s %45s\n","Hits", "IP","Hostname");
    my $count = 0;
    foreach my $ip (sort {$h{$b} - $lh{$b} <=> $h{$a} - $lh{$a} }  keys %h) {
        if ($top > $count++) {
            print " "x$width . "\r";  # clear line
            print colored ['bold white'], sprintf("%5i ",$h{$ip}),
                  colored ['bold green'], sprintf("+%-4i",$h{$ip} - $lh{$ip}),
                  colored ['red'], sprintf("%17s",$ip),
                  colored ['yellow'], sprintf(" %45s",&hostname($ip)),
                  "\n";
        } elsif ($count > 1000) {
            delete($h{$ip});
        }
    }

    # Print out the 'most popular' lookups
    $count = 0;
    my $half = floor($width/2);
    my $halfminus1 = $half-1;
    print colored ['bold black on white'], sprintf("%-${halfminus1}s\n", "Total Hits      Request");
    foreach my $req (sort {$r{$b} <=> $r{$a}} keys %r) {
        if ($top > $count++) {
            print " "x$width . "\r";
            print colored ['bold white'], sprintf("%5i ",$r{$req}),
                  colored ['bold green'], sprintf("+%-4i",$r{$req} - $lr{$req});
            #$req =~ s/(.{$half}).*/$1/,
            print colored ['bold blue'], $req, "\n";
        } elsif ($count > 1000) {
            delete($r{$req});
        }
    }
    return if ($width < 70);
    $count = 0;

    # Print out the 'most popular' lookups for this last data cycle
    print up($top+1), right($half);
    print colored ['bold black on white'], sprintf("%-${half}s\n", "New Hits      Request");
    foreach my $req (sort {$r{$b} - $lr{$b} <=> $r{$a}- $lr{$a} } keys %r) {
        if ($top > $count++) {
            print right($half),
                  colored ['bold green'], sprintf("+%-4i",$r{$req} - $lr{$req});
            my $max_string_size = $half-5;
            $req =~ s/(.{$max_string_size}).*/$1/,
            print colored ['bold blue'], $req, "\n";
        }
    }
    %lr = {};
}

sub hostname {
    my $ip = shift;
    $dns{$ip} ||= gethostbyaddr(inet_aton($ip), AF_INET) || 'No.rDNS';
    return $dns{$ip};
}

sub getStats {
    # get clients from 'rndc' ouput and use 'ps' to get CPU usuage of bind
    open RNDC, "rndc status |" or return undef;
    local $/ = undef;
    $_ = <RNDC>;
    close RNDC;
    $stats{cpu} = `ps axo %cpu,comm | grep named`;
    $stats{cpu} =~ s/ named.*/%/s;
    /version: BIND (\S+)/s || return undef;
    $stats{version} = $1;
    if (m,(recursive clients: \d+/\d+/\d+),s) {
        $stats{clients} = $1;
    }
}

sub header {
    # stuffed this in a subroutine so you can change it to what you want!
    my $now_string = localtime;
    print colored ['black on yellow'], sprintf("%-${width}s","$hostname    $now_string");
}

sub footer {
    print " "x$width . "\r";  # clear line
    my $size = $width - length("CPU: $stats{cpu}, BIND $stats{version}") - 1;
    my $now_string = localtime;
    print colored ['black on yellow'], sprintf("CPU: $stats{cpu}, BIND $stats{version} %${size}s\n",$stats{clients});
}
 
This tmux config (.tmux.conf) will ensure that the status bar is only visible if you have more than one window.

Code:
set-option -g status off

set-hook -g session-window-changed \
  'if-shell \
    "test $(tmux list-windows | wc -l) -eq 1" \
    "set-option status off" "set-option status on"'
 
I recently wrote a little Python script to display various CPU graphs (frequency, temperature, load) in a window on my X11 desktop.
It might be useful for others, too, so I put it on this web page. There are also screen shots.
The script may also serve as an example how to write simple X11 applications in Python. It also demonstrates how to retrieve and handle sysctl values in Python on FreeBSD.
Please read the instructions on the web page carefully. In particular, you have to install a few dependencies (Python3 for the script, Pillow for the image handling, Tkinter for the X11 widgets).
 
This may seem trivial, but it took longer to figure out than anticipated, so, why not share it?

Code:
#!/bin/sh
# keycodes - this checks out on FreeBSD 13.0, Debian 10.9, Ubuntu 20.04, LM 20.1
# 2021-05-17 m 099/jch

# Notes: dd returns "" for both null and lf (ctrl-@ and ctrl-J). This script
# returns "nl" (newline) for both. Ctrl-@ represents byte code 0 (ascii "nul").
# Ctrl-J nicknames: linefeed (lf) = newline (nl) = dec 10 = octal 012 = hex 0a.
# TERM values tested: xterm, xterm-256color, linux
# Emulations tested: Konsole, "xterm", MATE-terminal, FreeBSD &GNU virtual terms

#--- what syntax style does this shell implementation use?
if [ "$(echo '\40')" = " " ]; then shsyntax=older
elif [ $'\177' = $'\x7f' ]; then shsyntax=newer
else shsyntax=unfamiliar
fi
#--- initialize variables
case "$shsyntax" in
  ("older") nul='\000'; lf='\012'; del='\177'
    ;;
  ("newer") nul=$'\000'; lf=$'\012'; del=$'\177'
    ;;
  (*)
    echo; echo "Unfamiliar /bin/sh syntax. Bailing out, sorry."; echo
    exit 1
    ;;
esac
#--- let's get it started
cat << EOF
Press ENTER to exit, or any other key to see key codes and sequences.
(Some keys may be intercepted by the OS, terminal, or terminal emulation.)
EOF
notraw=$(stty -g)
stty raw
stty -echo
saveifs=$IFS
IFS=
loop=1
while [ "$loop" ]; do 
  key=$(dd bs=1 count=1 2>/dev/null) # bytes out on std oput, std err=info+error
  if [ "$key" = "" ]; then key=$lf; fi
  #--- check for control characters and whitespace
  if [ "$(echo "$key" | tr -d [:cntrl:])" = "" ]||[ "$key" = " " ]; then
    #--- look up human readable ascii nicknames for control chars and spacebar
    asc=$(echo -n "$key" | od -a | awk 'BEGIN{FS=" "}{print $2}')
  else #--- not a control char or " "
    asc=$key #--- this key char should be displayable as is
  fi
  echo -n $asc" " #--- display the character or it's nickname
  if [ "$asc" = "cr" ]; then loop= ; fi
done
IFS=$saveifs
stty "$notraw"
stty echo
echo #--- start new line on terminal
exit 0
 
Same as in previous post, but improved to handle utf-8 multibyte characters. Also followed some of olli@'s advice from post #333. The bits about sh syntax are just there to make it work properly on GNU implementations of the "sh" shell, which are not as well caught up to the more recent posix standards. This might also allow it to work on older FreeBSD sh versions.
Code:
#!/bin/sh
# keycodes - this is working on FreeBSD 13.0, Debian 10.9, Ubuntu 20.04, LM 20.1
# 2021-05-17 m 099/jch

# Notes: dd returns "" for both null and lf (ctrl-@ and ctrl-J). This script
# returns "nl" (newline) for both. Ctrl-@ represents byte code 0 (ascii "nul").
# Ctrl-J nicknames: linefeed (lf) = newline (nl) = dec 10 = octal 012 = hex 0a.
# TERM values tested: xterm, xterm-256color, linux
# Emulations tested: Konsole, "xterm", MATE-terminal, FreeBSD &GNU virtual terms

set -efu # errexit, noglob, nounset
myname=${0##*/}
shsyntax=
lf=
notraw=
saveifs=
loop=
byt=
hex=
utf8ch=
ucount=
uexpect=
asc=

#--- what syntax style does this shell implementation use?
if [ "$(echo '\40')" = " " ]; then shsyntax=older
elif [ $'\177' = $'\x7f' ]; then shsyntax=newer
else shsyntax=unfamiliar
fi
#--- initialize variables
case "$shsyntax" in
  ("older") lf='\012'
    ;;
  ("newer") lf=$'\012'
    ;;
  (*)
    echo; echo "Unfamiliar /bin/sh syntax. Bailing out, sorry."; echo
    exit 1
    ;;
esac
#--- let's get it started
cat << EOF
$myname - Press ENTER to exit, or any other key to see key codes and sequences.
(Some keys may be intercepted by the OS, terminal, or terminal emulation.)
EOF
notraw=$(stty -g)
stty raw
stty -echo
saveifs=$IFS
IFS=
ucount=0
loop=1
while [ "$loop" ]; do
  byt=$(dd bs=1 count=1 2>/dev/null)
  #--- dd sends bytes out on std oput, std err gets informational oput + errors
  if [ "$byt" ]; then
    hex=$(echo $byt | od -t x1 | awk 'BEGIN{FS=" "}{print $2}') #--- hex
  fi
  if [ "$byt" ]&&[ "$hex" \> "7f" ]; then
    #--- utf8 multibyte sequence handler
    ucount=$(($ucount+1))
    case $ucount in
      (1)
        utf8ch=$byt
        if [ "$hex" \> "bf" ]&&[ "$hex" \< "e0" ]; then #--- c0..df leadin
          uexpect=2
        elif [ "$hex" \> "df" ]&&[ "$hex" \< "f0" ]; then #--- e0..ef leadin
          uexpect=3
        elif [ "$hex" \> "ef" ]&&[ "$hex" \< "f8" ]; then #--- f0..f7 leadin
          uexpect=4
        else
          echo -n $myname': Invalid utf8 leadin byte="x'$hex'" '
          ucount=0
        fi
        ;;
      (*)
        if [ "$hex" \> "7f" ]&&[ "$hex" \< "c0" ]; then #--- 80..bf contin byte
          utf8ch=$utf8ch$byt
          if [ $ucount -eq $uexpect ]; then
            echo -n $utf8ch" " #--- display the multi-byte utf8 character
            ucount=0 #--- reset counter for next multi-byte utf8 sequence
          fi
        else
          echo -n $myname': Invalid utf8 continuation byte="x'$hex'" '
          ucount=0
        fi
        ;;
    esac
  else
    #--- ascii or utf8 single-byte handler for bytes in hex [00..7f] range
    if [ "$byt" = "" ]; then byt=$lf; fi #--- either nul or lf -> lf
    if [ $ucount -gt 0 ]; then
      echo -n $myname': Invalid utf8 character sequence '
      ucount=0
    #--- check for control characters and whitespace
    elif [ "$(echo "$byt" | tr -d [:cntrl:])" = "" ]||[ "$byt" = " " ]; then
      #--- look up human readable ascii nicknames for control chars and spacebar
      asc=$(echo -n "$byt" | od -a | awk 'BEGIN{FS=" "}{print $2}')
    else #--- not a control char or " "
      asc=$byt #--- this key char should be displayable as is
    fi
    echo -n $asc" " #--- display the character or it's nickname
    if [ "$asc" = "cr" ]; then loop= ; fi
  fi
done
IFS=$saveifs
stty "$notraw"
stty echo
echo #--- start new line on terminal
exit 0
 
Just a tiny script to check your /etc/rc.conf for consistency with ifconfig_* variables when you have lots of interfaces:
Code:
#!/bin/sh

if=${1:-lo0}

. /etc/rc.subr
. /etc/network.subr

load_rc_config netif
echo -n "${if}     : "
ifconfig_getargs ${if}

if ipv6if ${if}; then
    echo -n "${if} (v6): "
    ifconfig_getargs ${if} ipv6
else
    echo no IPv6
fi
Example usage:
Code:
# ./rc_ifconfig.sh bridge1
bridge1     : inet 192.168.99.1 netmask 255.255.255.0 addm vtnet3 addm lagg0v100
bridge1 (v6): inet6 2001:<redacted>:99::1 prefixlen 80 auto_linklocal

You can check everything you configured manually for an interface is actually there. Duplicate variables aren't that easy to spot in large configs, I actually found one using that script (and the problem just didn't manifest because other custom scripts happened to correct the interface's config later)
 
I needed to grab the entire Debian i386 & amd64 repo and yet I didn't really feel like logging into a Linux system to do so. Instead I wrote this (fairly substantial) Awk script :)

C-like:
#!/usr/bin/awk -f

# Example sources.list
# deb file:///repodir/bullseye-security/non-free/amd64 ./

############################################################################
# main
############################################################################
function main()
{
  add_source("http://deb.debian.org/debian",
    "bullseye", "main contrib non-free", "i386 amd64")

  add_source("http://deb.debian.org/debian",
    "bullseye-updates", "main contrib non-free", "i386 amd64")

  add_source("http://deb.debian.org/debian-security",
    "bullseye-security", "main contrib non-free", "i386 amd64")

  fetch()
  verify()
}

############################################################################
# add_source
############################################################################
function add_source(url, dist, components, archs,    curr, sc, sa, c, a)
{
  split_whitespace(components, sc)
  split_whitespace(archs, sa)

  for(c in sc)
  {
    for(a in sa)
    {
      curr = ++ALLOC
      SOURCES[curr] = curr
      SourceUrl[curr] = url
      SourceDist[curr] = dist
      SourceComp[curr] = sc[c]
      SourceArch[curr] = sa[a]
      SourcePackageDir[curr] = dist "/" SourceComp[curr] "/" SourceArch[curr]
    }
  }
}

############################################################################
# verify
############################################################################
function verify(    source)
{
  for(source in SOURCES)
  {
    verify_packages(source)
  }
}

############################################################################
# fetch
############################################################################
function fetch(    source)
{
  for(source in SOURCES)
  {
    fetch_metadata(source)
  }

  for(source in SOURCES)
  {
    fetch_packages(source)
  }
}

############################################################################
# verify_packages
############################################################################
function verify_packages(source,    input, line, tokens, tc, filename, checksum)
{
  input = SourcePackageDir[source] "/Packages"
  filename = ""
  checksum = ""

  if(!exists(input))
  {
    return
  }

  while(getline line < input == 1)
  {
    tc = split_whitespace(line, tokens)

    if(tc >= 2)
    {
      if(tokens[0] == "Filename:")
      {
        filename = tokens[1]
      }
      else if(tokens[0] == "SHA256:")
      {
        checksum = tokens[1]
      }
    }

    if(filename != "" && checksum != "")
    {
      print("Verifying: " filename)

      if(!exists(SourcePackageDir[source] "/" filename))
      {
        error("Package does not exist")
      }

      if(sha256(SourcePackageDir[source] "/" filename) != checksum)
      {
        error("Package checksum did not match")
      }

      filename = ""
      checksum = ""
    }
  }

  close(input)
}

############################################################################
# fetch_packages
############################################################################
function fetch_packages(source,    input, line, output, tokens, tc, skip, filename, checksum, url)
{
  input = SourcePackageDir[source] "/Packages.orig"
  output = "Packages.part"
  filename = ""
  checksum = ""

  if(exists(SourcePackageDir[source] "/Packages"))
  {
    return
  }

  touch(output)

  while(getline line < input == 1)
  {
    skip = 0
    tc = split_whitespace(line, tokens)

    if(tc >= 2)
    {
      if(tokens[0] == "Filename:")
      {
        filename = tokens[1]
        skip = 1
        print("Filename: " basename(filename)) > output
      }
      else if(tokens[0] == "SHA256:")
      {
        checksum = tokens[1]
      }
    }

    if(!skip)
    {
      print(line) > output
    }

    if(filename != "" && checksum != "")
    {
      url = SourceUrl[source] "/" filename
      filename = basename(filename)

      if(!exists(SourcePackageDir[source] "/" filename))
      {
        download(url, SourcePackageDir[source] "/" filename, checksum)
      }
      else
      {
        print("Package exists [" filename "]")
      }

      filename = ""
      checksum = ""
    }
  }

  close(output)
  close(input)

  mv("Packages.part", SourcePackageDir[source] "/Packages")
  rm(SourcePackageDir[source] "/Packages.orig")
}

############################################################################
# fetch_metadata
############################################################################
function fetch_metadata(source,    dir)
{
  dir = SourcePackageDir[source]

  if(exists(dir "/Packages"))
  {
    return
  }

  if(exists(dir "/Packages.orig"))
  {
    return
  }

  download(SourceUrl[source] "/dists/" SourceDist[source] "/" SourceComp[source] "/binary-" SourceArch[source] "/Packages.xz", "Packages.xz")

  if(system("xz -d 'Packages.xz'") != 0)
  {
    error("Failed to decompress meta-data")
  }

  mkdir_p(dir)
  mv("Packages", dir "/Packages.orig")
}

############################################################################
# rm
############################################################################
function rm(path)
{
  if(system("rm '" path "'") != 0)
  {
    error("Failed to remove file")
  }
}

############################################################################
# mv
############################################################################
function mv(source, dest)
{
  if(system("mv '" source "' '" dest "'") != 0)
  {
    error("Failed to move file")
  }
}

############################################################################
# mkdir_p
############################################################################
function mkdir_p(path)
{
  if(system("mkdir -p '" path "'") != 0)
  {
    error("Failed to create diectory")
  }
}

############################################################################
# error
############################################################################
function error(message)
{
  print("Error: " message)
  exit(1)
}

############################################################################
# sha256
############################################################################
function sha256(path,    cmd, line)
{
  cmd = "sha256 -q '" path "'"
  #cmd = "sha256sum '" path "' | awk '{ print $1 }'"

  if(cmd | getline line != 1)
  {
    error("Failed to generate checksum")
  }

  close(cmd)

  return line
}

############################################################################
# download
############################################################################
function download(source, dest, checksum,    fetch_cmd)
{
  #fetch_cmd = "ftp -o"
  #fetch_cmd = "wget -O"
  fetch_cmd = "fetch -qo"

  print("Fetching: " basename(source))

  if(system(fetch_cmd " 'download.a' '" source "'") != 0)
  {
    error("Failed to download")
  }

  if(!checksum)
  {
    if(system(fetch_cmd " 'download.b' '" source "'") != 0)
    {
      rm("download.a")
      error("Failed to download")
    }

    if(sha256("download.a") != sha256("download.b"))
    {
      rm("download.a")
      rm("download.b")
      error("Checksums do not match")
    }

    rm("download.b")
  }
  else
  {
    if(sha256("download.a") != checksum)
    {
      rm("download.a")
      error("Checksums do not match")
    }
  }

  mv("download.a", dest)
}

############################################################################
# exists
############################################################################
function exists(path)
{
  if(system("test -e '" path "'") == 0)
  {
    return 1
  }

  return 0
}

############################################################################
# touch
############################################################################
function touch(path)
{
  if(system("touch '" path "'") != 0)
  {
    error("Failed to touch file")
  }
}

############################################################################
# basename
############################################################################
function basename(path,    ci, ls)
{
  ls = -1

  for(ci = 1; ci <= length(path); ci++)
  {
    if(substr(path, ci, 1) == "/")
    {
      ls = ci
    }
  }

  if(ls == -1) return path

  return substr(path, ls + 1)
}

############################################################################
# split_whitespace
#
# Split the string by any whitespace (space, tab, new line, carriage return)
# and populate the specified array with the individual sections.
############################################################################
function split_whitespace(line, tokens,    curr, c, i, rtn)
{
  rtn = 0
  curr = ""
  delete tokens

  for(i = 0; i < length(line); i++)
  {
    c = substr(line, i + 1, 1)

    if(c == "\r" || c == "\n" || c == "\t" || c == " ")
    {
      if(length(curr) > 0)
      {
        tokens[rtn] = curr
        rtn++
        curr = ""
      }
    }
    else
    {
      curr = curr c
    }
  }

  if(length(curr) > 0)
  {
    tokens[rtn] = curr
    rtn++
  }

  return rtn
}

BEGIN { main() }
 
Sleep (suspend)

Sleep daemon for monitoring battery life and putting laptop to sleep when needed

ZFS

Can a generic script take cache vdevs (physical devices) offline?

<https://openzfs.github.io/openzfs-docs/man/8/zpool-offline.8.html>

Audio, ZFS and sleep

For now, part of the sleep-related script at <https://forums.freebsd.org/posts/531858> uses non-generic labels to identify the devices to be off-lined:

Code:
# Prepare to disconnect USB flash drives that are used for L2ARC
#
# zpool offline copperbowl gpt/cache-copperbowl
zpool offline august gpt/cache-august
zpool offline august gpt/duracell
sync
 
I finally got around to posting up my python-based shell-column-colorizer. A picture is really best here:

colcolor.png


Be one of the first to try it out. No installer as it's just a stand-alone (py3) script. If there's interest a very trivial port could be put together.

Happy to take pull requests!
 
When 'gcc -M' dependency generation (between the object files and the C header files) does not quite fit, I use my own script which does not count the standard C headers (I'm not going to edit them).
Actually, I have two similar scripts - one for UNIXes and one for Windows.

for UNIX:
Code:
use Cwd;
use Cwd 'abs_path';
use File::Basename;

my @includes=();
while(@ARGV and $ARGV[0] =~ /^-/){
    $_=shift;
    if (/^-(iquote|I)(.+)/){my @dirs=split(/:/,$2); push @includes, @dirs}
}

my $obj_file='';
my $dep_file='';

if(@ARGV){$obj_file= shift @ARGV}
if(@ARGV){$dep_file= shift @ARGV}
if(!$obj_file || !$dep_file){die "Not enough arguments"}
if(@ARGV){die "Too many arguments"}

my $source_file = fileparse($obj_file,qr/\.o/);
my $dir = dirname($obj_file);
my $cpp=$dir.'/'.$source_file.'.cpp';
my $c=$dir.'/'.$source_file.'.c';
my $scr=$dir.'/'.$source_file.'.scr'; # C++

if ( -e $cpp) {$source_file=$cpp} elsif ( -e $c) {$source_file=$c} elsif ( -e $scr) {$source_file=$scr} else {die "Source file not found"}

my @lines;
if(open(F_INP, $source_file)){
    @lines=<F_INP>;
    close(F_INP);
}

unless(open(F_OUT,'>>',$dep_file)){die "Could not open output file"}

sub find_first_dir_with_include_file {
    my $full_name='';
    for my $i (@includes){
        $full_name=abs_path($i.'/'.$_[0]);
        if( -e $full_name){last} else {$full_name='';}
    }
    return $full_name;
}

my $flag=0;
while(@lines){
    $line= shift @lines;
    if($line =~ /^s*#\s*include\s+"(.+)"/){
        if(!$flag){
            print F_OUT '$(_TMP_DIR)/'.$obj_file.': '.$source_file;
            $flag=1
        }
        my $name=$1;
        unless($name=~/\//){
            $name = dirname($source_file).'/'.$name;
        }
        print F_OUT ' '.&find_first_dir_with_include_file($name);
    }
}
print F_OUT "\n";
close(F_OUT);

for Windows:
Code:
use Win32;
use File::Basename;

my @includes=();
my $pch_file='';
while(@ARGV and $ARGV[0] =~ /^-/){
    $_=shift;
    if         (/^-p(.+)/){$pch_file=$1}
    elsif    (/^-(iquote|I)(.+)/){my @dirs=split(/;/,$2); push @includes, @dirs}
}

my $obj_file='';
my $dep_file='';

if(@ARGV){$obj_file= shift @ARGV}
if(@ARGV){$dep_file= shift @ARGV}
if(!$obj_file || !$dep_file){die "Not enough arguments"}
if(@ARGV){die "Too many arguments"}

fileparse_set_fstype( "MSDOS" );
my $source_file = fileparse($obj_file,qr/\.obj/);
my $dir= dirname($obj_file);
my $cpp=$dir.'/'.$source_file.'.cpp';
my $c=$dir.'/'.$source_file.'.c';
my $scr=$dir.'/'.$source_file.'.scr'; # C++

if ( -e $cpp) {$source_file=$cpp} elsif ( -e $c) {$source_file=$c} elsif ( -e $scr) {$source_file=$scr} else {die "Source file not found"}

my @lines;
if(open(F_INP, $source_file)){
    @lines=<F_INP>;
    close(F_INP);
}

unless(open(F_OUT,'>>',$dep_file)){die "Could not open output file"}

sub find_first_dir_with_include_file {
    my $full_name='';
    for my $i (@includes){
        $full_name=Win32::GetFullPathName($i.'/'.$_[0]);
        if( -e $full_name){last} else {$full_name='';}
    }
    return $full_name;
}

my $flag=0;
while(@lines){
    $line= shift @lines;
    if($line =~ /^s*#\s*include\s+"(.+)"/){
        if(!$flag){
            if($pch_file){
                print F_OUT $pch_file.': '.$source_file;
            } else {
                print F_OUT '$(_TMP_DIR)/'.$obj_file.': '.$source_file;
            }
            $flag=1
        }
        my $name=$1;
        if($name!~/\// && $name!~/\\/) {
            $name = dirname($source_file).'/'.$name;
        }
        print F_OUT ' '.&find_first_dir_with_include_file($name);
    }
}
print F_OUT "\n";
close(F_OUT);

I use them in makefiles as follows:
Code:
ifeq ($(PLATFORM),WINDOWS)
 _MAKEDEP_UTIL:=perl.exe $(UTIL_DIR)/makedep_win.pl
else
 _MAKEDEP_UTIL:=$(UTIL_DIR)/makedep_lin.pl
endif

and then:
Code:
ifeq ($(PLATFORM),WINDOWS) # [*]
define _CallMakeDepPCH
@$(_MAKEDEP_UTIL)  -p$(_PCH_NAME) $(_INCLUDES) $(USE_PCH) $(_PROJ_DEP)
endef
endif # [*]
define _CallMakeDep
@$(_MAKEDEP_UTIL)  $(_INCLUDES) $(1) $(_PROJ_DEP)
endef

and finally:
Code:
dep:
    $(foreach src,$(_OBJECTS),$(call _CallMakeDep, $(src)))
ifdef USE_PCH
    $(call _CallMakeDepPCH)
endif

Looks boring , but works
 
Last edited:
Back
Top