Useful scripts

Mjölnir

Daemon

Reaction score: 1,463
Messages: 2,114

Hi people!
Please, post here useful scripts which can help to do some useful or difficult operation on FreeBSD OS.
Found a script on one of my old laptops to attach/insert a geom(4) I/O scheduler - RTFM gsched(8).
  • An I/O scheduler is at least useful on a UFS (or alike) system; I don't know and didn't try with ZFS. Other use cases might be a DB on a dedicated geom(4) device.
  • The I/O scheduler can significantly increase throughput and interactivity with concurrent tasks, as described by the authors of gsched(8).
    I can could verify that - the system runs much "smoother" when running background tasks with heavy disk I/O. The script served me well for years.
  • The script might violate some guidelines for rc scripting that I was not aware of.
  • Very Likely it has bugs, since I was the only user.
    Drop me a note and I'll fix that.
  • If a few others could prove it's safe, it could go to /usr/share/examples/etc/rc.d or even in the base.
  • Suggested directories:{/usr/local}/etc/rc.d/geom_sched, {/usr/local}/libexec/fs_summarize.awk
    or the respective directories in /usr/share/examples
    and rc.geom_sched.conf is for insertion in rc.conf{.local}
The other script fs_summarize.awk is useful to determine the parameters for newfs(8) or just if you want some info about a FS or directory.
Just in case this is needed: the standard FreeBSD BSD license applies to the scripts, the snippet rc_geom_sched.conf obviously does not need any license.
 

Attachments

  • rc_geom_sched.conf
    762 bytes · Views: 150
  • geom_sched.txt
    4.8 KB · Views: 114
  • fs_summarize.awk.txt
    4.7 KB · Views: 77

kpedersen

Daemon

Reaction score: 1,566
Messages: 2,422

Moving UNIX domain sockets
Not really a script as such but more an example that UNIX domain sockets can be moved (not copied). This is really handy for allowing X11 access into a Jail.

Code:
UNIQUE=$$
Xephyr :$UNIQUE &

# Wait until the socket exists

mv /tmp/.X11-unix/X$UNIQUE /jails/myjail/tmp/.X11-unix/X0
jexec myjail xterm -display :0

Finding absolute path of script or program
Also, a handy way to get the absolute path to your script (or more usefully, the prefix) without faffing with procfs and all that other stuff. This is useful because sometimes relative paths are not supported or not desired.

Code:
PREFIX="$(cd "$(dirname "$(which "$0")")" && cd .. && pwd)"

# which "$0" ensures that a path is returned if the program was run from $PATH
# dirname ensures that the directory containing the program (i.e bin) is returned
# cd, cd .., pwd ensures that an absolute path is given by going into the directory and using pwd.

It seems a bit "dirty" but I also use this (via popen) in C++ and perl. It is the only guaranteed way that I have found over the years. And I have tried quite a few!
 

olli@

Daemon
Developer

Reaction score: 1,205
Messages: 1,108

Not really a script as such but more an example that UNIX domain sockets can be moved (not copied). This is really handy for allowing X11 access into a Jail.

Code:
[...]
mv /tmp/.X11-unix/X$UNIQUE /jails/myjail/tmp/.X11-unix/X0
Does that also work if /tmp and /jails are on different file systems? Because in that case a you’re technically copying, not moving.
Also, a handy way to get the absolute path to your script (or more usefully, the prefix) without faffing with procfs and all that other stuff. This is useful because sometimes relative paths are not supported or not desired.

Code:
PREFIX="$(cd "$(dirname "$(which "$0")")" && cd .. && pwd)"
The realpath(3) function (and the corresponding realpath(1) command) is very useful in this context. It converts relative paths to absolute paths and resolves symbolic links. In shell scripts I'm using the following to find out the script’s directory:
Code:
SCRIPTDIR=$(realpath "${0%/*}")
So, to get the prefix, that would be rather simple:
Code:
PREFIX=$(realpath "${0%/*}/..")
Sometimes I resort to writing a script in zsh, because FreeBSD's sh is missing quite a few features (it’s not a POSIX-compatible shell). In that case, the prefix can be found without calling any external programs, so this is most efficient:
Code:
PREFIX=${0:A:h:h}
Explanation: Basically, the “:A” modifier is equivalent to realpath(3), and the “:h” modifier is equivalent to dirname(3).

By the way, in Python you would do this:
Code:
from sys import argv
from os.path import realpath, dirname

prefix = dirname(dirname(realpath(argv[0])))
Other languages like Perl or Ruby also have bindings to the standard library functions realpath(3) and dirname(3), so it basically works the same there, too (modulo different syntax, of course). C and C++ can use those functions directly anyway, of course – the former is in <stdlib.h> and the latter is in <libgen.h>. Both are POSIX / SUS standards.
 

kpedersen

Daemon

Reaction score: 1,566
Messages: 2,422

Does that also work if /tmp and /jails are on different file systems? Because in that case a you’re technically copying, not moving.
Unfortunately no, I don't think that is possible. Perhaps with something like nullfs? You can trick it into opening the socket on the other FS?

Perhaps it is time for Zirias's project to come in handy? :): https://forums.freebsd.org/threads/new-tool-remusock-remote-unix-socket-access.76026/


The realpath(3) function (and the corresponding realpath(1) command) is very useful in this context. It converts relative paths to absolute paths and resolves symbolic links.

From what I could see, this doesn't work for things in PATH? I.e realpath ls returns /home/kpedersen/ls
 

olli@

Daemon
Developer

Reaction score: 1,205
Messages: 1,108

From what I could see, this doesn't work for things in PATH? I.e realpath ls returns /home/kpedersen/ls
Yes, it does work for things in PATH. When you call a shell script via PATH, $0 always includes the directory in which it was found.
 

olli@

Daemon
Developer

Reaction score: 1,205
Messages: 1,108

By the way, some generic advice for writing shell scripts, for those wo are not already experts … ;)

Try to write shell scripts using /bin/sh only. This will make the scripts most portable and doesn't require people to install a specific shell if you share them with others. Note that /bin/sh scripts will usually work without change with POSIX-compatible shells, too (zsh, ksh, bash). If you absolutely have to use POSIX features that are not supported by FreeBSD's /bin/sh, I suggest using ksh because it most closely resembles the POSIX standard, and also works with bash and zsh. Avoid using bashisms or zshisms that don't work anywhere else.

Begin your script with set -Cefu.
  • -C (“noclobber”) prevents the script from overwriting files with > accidentally. If you need to overwrite a file on purpose and you're absolutely sure that it’s the correct thing to do, you can use >| instead to override the -C option, or enclose the respective code between set +C and set -C, or simply rm -f the file beforehand.
  • -e (“errexit”) causes the script to terminate immediately if a command returns an error (i.e. a non-zero return code). See the sh(1) manual page for details. If you know that a command may return a non-zero exit code, but you want to continue anyway, append || true, or (better) put it into an if clause and print a message to the user if appropriate.
  • -f (“noglob”) disables globbing, also known as pathname expansion or filename generation using wildcards like *. This is often a cause of subtle bugs and malfunctions, so it’s best to disable it altogether. To create a list of files, it’s better to use things like ls | grep ... or find(1) which is much more powerful.
  • -u (“nounset”) causes the script to print an error message and exit immediately if you try to use a variable that has not been set before. unset FOO; echo $FOO will trigger this, for example. This is useful to quickly detect typos in variable names. If you need to expand a variable that might be unset (and you known what you’re doing), ${FOO-} will prevent the error (if FOO is unset, it will expand to an empty string). Again, see the sh(1) manual page for details.
These options are very helpful to write scripts that are more robust and have a lower risk of malfunctions.

The next line in most of my scripts is: ME=${0##*/}
That sets the variable ME to the basename of the script, so it can conveniently be used in error messages and similar, like in this little helper function:
Code:
Err ()
{
        echo "${ME}: $*" >&2
        exit 1
}
So you can write things like this:
Code:
test -f "$INFILE" || Err "File not found: $INFILE"

If in doubt, always write variable expansions in double quotes.
If a variable might begin with a dash, use -- to prevent it from being confused with an option if appropriate, e.g.:
ls -l -- "$MYDIR"

By the way, do not use backticks ( `foo`) for command substitution. Nesting them is difficult, they can be confused with single quote characters ( 'foo' looks very similar), and the quoting rules involving backticks are complicated. Better use the alternative $(foo) syntax. This can be nested arbitrarily deep without having to use heaps of backslashes, and the quoting rules are more intuitive.

If you need to debug a shell script, the -v and -x options are very useful: sh -vx ./myscript
  • -v causes the script code to be printed as it is read from the file during execution.
  • -x causes the resulting commands (after variable expansion etc.) to be printed right before they're executed. This is especially useful to see what your script is actually doing.

Happy hacking!
 

kpedersen

Daemon

Reaction score: 1,566
Messages: 2,422

  • -u (“nounset”) causes the script to print an error message and exit immediately if you try to use a variable that has

That is so very useful for constructs like:

rm -rf "${LOGDIR}/${DATE}"

Where you just happen to leave those variables unassigned in an incorrect code-path. XD
 

decuser

Active Member

Reaction score: 74
Messages: 214

Sleep daemon for monitoring battery life and putting laptop to sleep when needed

Here's a useful set of scripts for creating a daemon to monitor battery life and sleep the system when it's critically low that works without a desktop (it also works with TWM, but it's not required). The original discussion on this topic is here, the workhorse is the monitor, which is a modified version of the script DutchDaemon originally posted here. The script basically checks the battery level every 5 minutes and when it drops to 10%, it sets a 2 minute timer and checks again, if it's getting lower at that time, it sends an email to root warning of imminent shutdown and 2 minutes after that sleeps the system.

I use the daemon because I have not been using a DE lately and my laptop kept powering down when the battery got low. This keeps that from happening.

The zipfile - sleepd.zip contains these three files:

1. rc-sleepd - the rc script to control the daemon, gets installed in /etc/rc.d
2. sbin-sleepd - the script that does the actual monitoring, gets installed in /usr/sbin
3. install.sh - a simple install script to install the daemon and remind you how to enable the service

To install, just download the zipfile as a normal user with wheel privileges and:

unzip sleepd.zip
cd sleepd
sudo sh ./install.sh


and

sudo sysrc -f /etc/rc.conf sleepd_enable="YES"
sudo service sleepd start


20200722-1418 wds - updated the scripts to use /usr/local prefixes in line with hier()
 

Attachments

  • sleepd.zip
    1.4 KB · Views: 47

kpedersen

Daemon

Reaction score: 1,566
Messages: 2,422

OpenSSH on Windows is kinda bearable compared to remote powershell. The biggest issue is no native tmux / screen support.
I was not happy with the existing practice of using a tabbed console or tmux on the local host and opening multiple ssh connections so I made my own tool.


Basically run bin/vimux and then you can use tmux-like shortcuts like ctrl-c to create new terminals within the same session.

(the underlying platform enabling this is vim and its virtual-pty support ;)
 

olli@

Daemon
Developer

Reaction score: 1,205
Messages: 1,108

In another thread I mentioned this shell function, and mjollnir asked me to put it here, so here it is.
It uses zsh syntax and is meant to be put in your ~/.zshrc, but I guess it could be adapted to bash without much effort.

Alternatively you can turn it into a standalone script: Remove the first line “inplace ()” and the outermost level of braces, and prepend the line “#!/usr/bin/env zsh”. Save the script with the name inplace in some bin directory (so it can be found via your $PATH setting), and make it executable: chmod 755 inplace. Make sure you have shells/zsh installed.
Code:
inplace ()
{
        #   Execute a command that reads from a file and writes to stdout,
        #   and move the output back to the input file.  The file name must
        #   be the last argument.  Timestamps (atime, mtime) and permissions
        #   are preserved.  A backup of the original file is retained with the
        #   extension ".BAK".  If an error occurs, the orignal file is not
        #   changed, and a temporary file is kept in the same directory
        #   (unless it is empty).

        emulate -L zsh
        local LASTARG="${@[-1]}"
        local BAKNAME="$LASTARG".BAK
        local TMPNAME="$LASTARG".inplace_tmp

        echo "Note: Keeping backup in $BAKNAME" >&2
        rm -f -- "$TMPNAME" \
        && "$@" > "$TMPNAME" \
        && touch -r "$LASTARG" -- "$TMPNAME" \
        && chmod 0$(stat -f"%OLp" -- "$LASTARG") "$TMPNAME" \
        && mv -f -- "$LASTARG" "$BAKNAME" \
        && mv -f -- "$TMPNAME" "$LASTARG" \
        || {
                if [[ ! -s "$TMPNAME" ]]; then
                        rm -f -- "$TMPNAME"
                fi
                return 1
        }
}
Usage example:
inplace iconv -f ISO8859-16 -t UTF-8 somefile.txt
 

Neubert

Member

Reaction score: 19
Messages: 30

Here's my current backup script in case it helps other new users with similar needs.

All of my data from multiple computers fits on a single hard drive, so I back it all up to a central FreeBSD server with a removable drive tray and store the drives offsite.

The morning protocol takes about 15 seconds; if the programmable LED is flashing green, I open the tray and swap the drive, then take it with me when I leave. At night I bring home an older backup drive and then repeat the swap procedure the next morning.

The script runs zfs scrub on insertion every Saturday, so all of the backup drives get scrubbed for errors once every few weeks as they rotate through.

I'm currently backing up a Windows desktop, an Ubuntu laptop, a FreeBSD server, and the backup server itself to the external drive.

The system detects drive insertion using this devd config:

Bash:
# cat /usr/local/etc/devd/backup.conf
notify 100 {
    # insert backup drive
    match "system" "GEOM";
    match "subsystem" "DEV";
    match "type" "CREATE";
    match "cdev" "gpt/backup";
    action "su -l backup -c 'doas /usr/local/sbin/backup.sh --on-insert' &";
};

notify 100 {
    # eject backup drive
    match "system" "GEOM";
    match "subsystem" "DEV";
    match "type" "DESTROY";
    match "cdev" "gpt/backup";
    action "su -l backup -c 'doas /usr/local/sbin/backup.sh --on-eject' &";
};

The script runs using doas for a dedicated user with this config:

Bash:
# cat /usr/local/etc/doas.conf
permit nopass backup as root cmd /usr/local/sbin/backup.sh

The backup script itself is attached below.
 

Attachments

  • usr-local-sbin-backup.zip
    1.8 KB · Views: 32
Last edited:

Rudy

Member

Reaction score: 6
Messages: 50

Here is a script to run in the terminal to give you some info on what your named server is doing.

Perl:
#!/usr/local/bin/perl
#
# Quick script to see that the DNS server is doing
# Fri Oct 30 16:34:29 PDT 2020, created, rudy
#
# requirements:
# pkg install p5-Term-ReadKey
# pkg install p5-Term-ANCIScreen
# ----------------------------------------------------------------------

use Term::ReadKey;
use Socket;
use POSIX qw(ceil floor);
use Term::ANSIScreen qw/:color :cursor :screen :keyboard :constants/;
use strict;

our (%h, %r, %lh, %lr);   # hits, requests, and last for those two
our (%dns, %stats);       # DNS cache, and RDNC output
our ($width, $height);   # terminal size
our $hostname = `hostname`;
chomp($hostname);

# set 'network' to your DNS server's ip block
# example, your DNS server is 10.8.2.3 or 10.8.2.4 then use
our $network = '10.8.1.0/28';  # DNS server Range (used in TCPDUMP)
our $port = 53;                   # not sure this will ever change
our $countPerCycle = 1000;        # count flag for TCPDUMP

# ----------------------------------------------------------------------

print "Welcome, starting TCPdump to gather snapshot of current DNS requests.\nStandby ......\n\n";
while (1) {
    &tcpdump;
    my ($lastwidth, $lastheight) = ($width, $height);
    ($width, $height) = GetTerminalSize ();
    print cls() if ($height != $lastheight || $width != $lastwidth);  #wipe screen on resize
    locate 1, 1;  # move cursor to top left
    &getStats;
    &header;
    &top10;
    &footer;
}

sub tcpdump {
    %lr = %r;
    %lh = %h;
    open TCPDUMP, "tcpdump -n -c $countPerCycle dst port $port and dst net $network 2> /dev/null |" or die "No tcpdump\n";
    while (<TCPDUMP>) {
        # 13:14:18.664904 IP 52.119.118.12.42467 > 208.69.40.3.53: 26522+ A? l3.dca0.com. (29)
        # this could probably use refinements... version 0.1
        my @a = split(' ');
        my $ip = $a[2];
        $ip =~ s/(\.\d+)$//; #remote port;
        $h{$ip}++; # easy to spoof...
        if (/Flags/) {
            $r{'Flags'}++;
        } elsif ($a[6] eq '[1au]') {
            $r{"$a[7] $a[8]"}++;
        } else {
            $r{"$a[6] $a[7]"}++;
        }
    }
    close TCPDUMP;
}


sub top10 {
    # started out as top10, but let's just max out to match terminal size.
    my $top =  floor($height/2) - 3;

    # print top IPs doing requests
    print colored ['black on white'], sprintf("%5s      %17s %45s\n","Hits", "IP","Hostname");
    my $count = 0;
    foreach my $ip (sort {$h{$b} - $lh{$b} <=> $h{$a} - $lh{$a} }  keys %h) {
        if ($top > $count++) {
            print " "x$width . "\r";  # clear line
            print colored ['bold white'], sprintf("%5i ",$h{$ip}),
                  colored ['bold green'], sprintf("+%-4i",$h{$ip} - $lh{$ip}),
                  colored ['red'], sprintf("%17s",$ip),
                  colored ['yellow'], sprintf(" %45s",&hostname($ip)),
                  "\n";
        } elsif ($count > 1000) {
            delete($h{$ip});
        }
    }

    # Print out the 'most popular' lookups
    $count = 0;
    my $half = floor($width/2);
    my $halfminus1 = $half-1;
    print colored ['bold black on white'], sprintf("%-${halfminus1}s\n", "Total Hits      Request");
    foreach my $req (sort {$r{$b} <=> $r{$a}} keys %r) {
        if ($top > $count++) {
            print " "x$width . "\r";
            print colored ['bold white'], sprintf("%5i ",$r{$req}),
                  colored ['bold green'], sprintf("+%-4i",$r{$req} - $lr{$req});
            #$req =~ s/(.{$half}).*/$1/,
            print colored ['bold blue'], $req, "\n";
        } elsif ($count > 1000) {
            delete($r{$req});
        }
    }
    return if ($width < 70);
    $count = 0;

    # Print out the 'most popular' lookups for this last data cycle
    print up($top+1), right($half);
    print colored ['bold black on white'], sprintf("%-${half}s\n", "New Hits      Request");
    foreach my $req (sort {$r{$b} - $lr{$b} <=> $r{$a}- $lr{$a} } keys %r) {
        if ($top > $count++) {
            print right($half),
                  colored ['bold green'], sprintf("+%-4i",$r{$req} - $lr{$req});
            my $max_string_size = $half-5;
            $req =~ s/(.{$max_string_size}).*/$1/,
            print colored ['bold blue'], $req, "\n";
        }
    }
    %lr = {};
}

sub hostname {
    my $ip = shift;
    $dns{$ip} ||= gethostbyaddr(inet_aton($ip), AF_INET) || 'No.rDNS';
    return $dns{$ip};
}

sub getStats {
    # get clients from 'rndc' ouput and use 'ps' to get CPU usuage of bind
    open RNDC, "rndc status |" or return undef;
    local $/ = undef;
    $_ = <RNDC>;
    close RNDC;
    $stats{cpu} = `ps axo %cpu,comm | grep named`;
    $stats{cpu} =~ s/ named.*/%/s;
    /version: BIND (\S+)/s || return undef;
    $stats{version} = $1;
    if (m,(recursive clients: \d+/\d+/\d+),s) {
        $stats{clients} = $1;
    }
}

sub header {
    # stuffed this in a subroutine so you can change it to what you want!
    my $now_string = localtime;
    print colored ['black on yellow'], sprintf("%-${width}s","$hostname    $now_string");
}

sub footer {
    print " "x$width . "\r";  # clear line
    my $size = $width - length("CPU: $stats{cpu}, BIND $stats{version}") - 1;
    my $now_string = localtime;
    print colored ['black on yellow'], sprintf("CPU: $stats{cpu}, BIND $stats{version} %${size}s\n",$stats{clients});
}
 

kpedersen

Daemon

Reaction score: 1,566
Messages: 2,422

This tmux config (.tmux.conf) will ensure that the status bar is only visible if you have more than one window.

Code:
set-option -g status off

set-hook -g session-window-changed \
  'if-shell \
    "test $(tmux list-windows | wc -l) -eq 1" \
    "set-option status off" "set-option status on"'
 

olli@

Daemon
Developer

Reaction score: 1,205
Messages: 1,108

I recently wrote a little Python script to display various CPU graphs (frequency, temperature, load) in a window on my X11 desktop.
It might be useful for others, too, so I put it on this web page. There are also screen shots.
The script may also serve as an example how to write simple X11 applications in Python. It also demonstrates how to retrieve and handle sysctl values in Python on FreeBSD.
Please read the instructions on the web page carefully. In particular, you have to install a few dependencies (Python3 for the script, Pillow for the image handling, Tkinter for the X11 widgets).
 
Top