general/other Debian/Ubuntu jail encounters GPG error in FreeBSD 14.0

I am using the same build script to create a Debian Bookworm jail compatibility layer. Everything works fine in 13.2, but in 14.0, I repeatedly encounter the error 'The repository InRelease' is not signed,' even when switching to official sources. I attempted to manually run 'apt --fix-broken install' and installed the GPG signature tool. However, these efforts were ineffective, and I now believe that the issue is being caused by FreeBSD. I found that this is likely a false error, and the actual issue might be caused by the filesystem of FreeBSD 14.0. The content of the script I am using is as follows:

Since many individuals have encountered similar issues in WSL and Docker, I have also checked the system's time, which is accurate.

Does anyone have any solutions?

Code:
#/bin/sh

rootdir=/compat/debian
baseurl="https://mirrors.ustc.edu.cn/debian/"
codename=bookworm

echo "begin to install Debian 12 AKA bookworm ..."
echo "check modules ..."

# check linux module
if [ "$(sysrc -n linux_enable)" = "NO" ]; then
        echo "linux module should be loaded. Continue?(N|y)"
        read answer
        case $answer in
                [Nn][Oo]|[Nn])
                        echo "linux module not loaded"
                        exit 1
                        ;;
                *)
                        sysrc linux_enable=YES
                        ;;
        esac
fi
echo "start linux"
service linux start

# check dbus
if ! /usr/bin/which -s dbus-daemon;then
        echo "dbus-daemon not found. install it [N|y]"
        read  answer
        case $answer in
            [Nn][Oo]|[Nn])
                echo "dbus not installed"
                exit 2
                ;;
            *)
                pkg install -y dbus
                ;;
        esac
    fi

if [ "$(sysrc -n dbus_enable)" = "NO" ]; then
        echo "dbus should be enable. Continue?(N|y)"
        read answer
        case $answer in
            [Nn][Oo]|[Nn])
                        echo "dbus not running"
                        exit 2
                        ;;
            *)
                        sysrc dbus_enable=YES
                        ;;
        esac
fi
echo "start dbus"
service dbus start

if ! /usr/bin/which -s debootstrap; then
        echo "debootstrap not found. install it? (N|y)"
        read  answer
        case $answer in
            [Nn][Oo]|[Nn])
                echo "debootstap not installed"
                exit 3
                ;;
            *)
                pkg install -y debootstrap
                ;;
        esac
    fi
echo "now we will bootstrap ${codename}.press any key."
read  answer

debootstrap ${codename} ${rootdir} ${baseurl}

if [ ! "$(sysrc -f /boot/loader.conf -qn nullfs_load)" = "YES" ]; then
        echo "nullfs_load should load. continue? (N|y)"
        read answer
        case $answer in
            [Nn][Oo]|[Nn])
                echo "nullfs not load"
                exit 4
                ;;
            *)
                sysrc -f /boot/loader.conf nullfs_load=yes
                ;;
        esac
    fi

if ! kldstat -n nullfs >/dev/null 2>&1;then
        echo "load nullfs module"
        kldload -v nullfs
fi

echo "mount some fs for linux"
echo "devfs ${rootdir}/dev devfs rw,late 0 0" >> /etc/fstab
echo "tmpfs ${rootdir}/dev/shm tmpfs rw,late,size=1g,mode=1777 0 0" >> /etc/fstab
echo "fdescfs ${rootdir}/dev/fd fdescfs rw,late,linrdlnk 0 0" >> /etc/fstab
echo "linprocfs ${rootdir}/proc linprocfs rw,late 0 0" >> /etc/fstab
echo "linsysfs ${rootdir}/sys linsysfs rw,late 0 0" >> /etc/fstab
echo "/tmp ${rootdir}/tmp nullfs rw,late 0 0" >> /etc/fstab
echo "/home ${rootdir}/home nullfs rw,late 0 0" >> /etc/fstab
mount -al

echo "NOW I will should change 'compat.linux.osrelease'. continue? (Y|n)"
read answer
case $answer in
    [Nn][Oo]|[Nn])
        echo "close to success"
        exit 4
        ;;
    [Yy][Ee][Ss]|[Yy]|"")
        echo "compat.linux.osrelease=6.2.10" >> /etc/sysctl.conf
        sysctl compat.linux.osrelease=6.2.10
                ;;
esac

echo "add ustc apt sources"
echo "deb http://mirrors.ustc.edu.cn/debian stable main contrib non-free non-free-firmware" > /compat/debian/etc/apt/sources.list
echo "# deb-src http://mirrors.ustc.edu.cn/debian stable main contrib non-free non-free-firmware" >> /compat/debian/etc/apt/sources.list
echo "deb http://mirrors.ustc.edu.cn/debian stable-updates main contrib non-free non-free-firmware" >> /compat/debian/etc/apt/sources.list
echo "# deb-src http://mirrors.ustc.edu.cn/debian stable-updates main contrib non-free non-free-firmware" >> /compat/debian/etc/apt/sources.list
echo "# deb http://mirrors.ustc.edu.cn/debian stable-proposed-updates main contrib non-free non-free-firmware" >> /compat/debian/etc/apt/sources.list
echo "# deb-src http://mirrors.ustc.edu.cn/debian stable-proposed-updates main contrib non-free non-free-firmware" >> /compat/debian/etc/apt/sources.list
echo "deb http://mirrors.ustc.edu.cn/debian-security/ stable-security main non-free contrib" >> /compat/debian/etc/apt/sources.list
echo "# deb-src http://mirrors.ustc.edu.cn/debian-security/ stable-security main non-free contrib" >> /compat/debian/etc/apt/sources.list
echo "Acquire::http::Pipeline-Depth \"0\";" > /compat/debian/etc/apt/apt.conf.d/99nopipelining
echo 'APT::Cache-Start "100000000";' | tee /compat/debian/etc/apt/apt.conf.d/70debcoebconf


echo "install nano fonts-wqy-microhei  fonts-wqy-zenhei and wget"
chroot ${rootdir} /bin/bash -c " apt update && apt --fix-broken install -y && apt upgrade && apt install nano wget fonts-wqy-microhei  fonts-wqy-zenhei -y"
echo "Now you can run '#chroot /compat/debian/ /bin/bash' Into debian"




Code:
root@ykla:/# apt  update
Get:1 https://mirrors.tuna.tsinghua.edu.cn/debian bookworm InRelease [151 kB]
Get:2 https://mirrors.tuna.tsinghua.edu.cn/debian bookworm-updates InRelease [52.1 kB]
Get:3 https://mirrors.tuna.tsinghua.edu.cn/debian bookworm-backports InRelease [54.7 kB]
Err:1 https://mirrors.tuna.tsinghua.edu.cn/debian bookworm InRelease
  At least one invalid signature was encountered.
Get:4 https://mirrors.tuna.tsinghua.edu.cn/debian-security bookworm-security InRelease [48.0 kB]
Err:2 https://mirrors.tuna.tsinghua.edu.cn/debian bookworm-updates InRelease
  At least one invalid signature was encountered.
Err:3 https://mirrors.tuna.tsinghua.edu.cn/debian bookworm-backports InRelease
  At least one invalid signature was encountered.
Err:4 https://mirrors.tuna.tsinghua.edu.cn/debian-security bookworm-security InRelease
  At least one invalid signature was encountered.
Reading package lists... Done
W: GPG error: https://mirrors.tuna.tsinghua.edu.cn/debian bookworm InRelease: At least one invalid signature was encountered.
E: The repository 'https://mirrors.tuna.tsinghua.edu.cn/debian bookworm InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: GPG error: https://mirrors.tuna.tsinghua.edu.cn/debian bookworm-updates InRelease: At least one invalid signature was encountered.
E: The repository 'https://mirrors.tuna.tsinghua.edu.cn/debian bookworm-updates InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: GPG error: https://mirrors.tuna.tsinghua.edu.cn/debian bookworm-backports InRelease: At least one invalid signature was encountered.
E: The repository 'https://mirrors.tuna.tsinghua.edu.cn/debian bookworm-backports InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: GPG error: https://mirrors.tuna.tsinghua.edu.cn/debian-security bookworm-security InRelease: At least one invalid signature was encountered.
E: The repository 'https://mirrors.tuna.tsinghua.edu.cn/debian-security bookworm-security InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
 
I see the same. debootstrap bullseye then apt update works OK on 13.1, but with this error after debootstrap on 14.0-ALPHA3.

Code:
# uname -a
FreeBSD freebsd14 14.0-ALPHA3 FreeBSD 14.0-ALPHA3 amd64 1400097 #0 stable/14-n265022-2af9390e54ed: Fri Aug 25 05:45:56 UTC 2023     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64

# pkg install sysutils/debootstrap
# kldload linux64 fdescfs linprocfs linsysfs tmpfs
# service linux onestart
# mkdir debian_build
# cd debian_build
# mkdir dev proc sys tmp
# mount -t linprocfs none `pwd`/proc
# mount -t devfs none `pwd`/dev
# mount -t linsysfs none `pwd`/sys
# mount -t tmpfs none `pwd`/tmp
# chmod 777 tmp
# chmod +t tmp
# debootstrap bullseye . http://deb.debian.org/debian
# chroot . /bin/bash

# apt update
Get:1 http://deb.debian.org/debian bullseye InRelease [116 kB]
Err:1 http://deb.debian.org/debian bullseye InRelease
  At least one invalid signature was encountered.
Reading package lists... Done
W: GPG error: http://deb.debian.org/debian bullseye InRelease: At least one invalid signature was encountered.
E: The repository 'http://deb.debian.org/debian bullseye InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

This is a workaround. Very unsafe when using the default HTTP mirror in /etc/apt/sources.list, so you should first replace that with a HTTPS mirror from here, checking certificate validity of that URL with curl -v first.

# apt -o Acquire::AllowInsecureRepositories=true -o Acquire::AllowDowngradeToInsecureRepositories=true update
# apt install git
 
Last edited by a moderator:
CURRENT isn't supported on the forums. There's a lot of reasons, but, generally, if you have something that is affecting you in CURRENT, you're better off asking on their mailing list.
 
Is there a sources.list for this particular configuration? After the deb, you could add [trusted=yes] which should allow installation despite the signing issue
 
After some digging around, it looks like this issue *might* be due to a difference between 13.x and 14.x in how the Linux binary emulation handles file attribute syscalls for non-root users inside chroots, specifically fgetxattr returning EPERM which prevents cp --preserve=mode from executing, and also causes errors in typical ls -l output.

It could also be something with my setup, but I am using clean VM installs. In any case, I'll check for bug reports and mailing list posts and file one if not already known.

---

Continuing from the debootstrap example above and applying this patch inside the chroot:

Code:
root@freebsd14:/# diff -Naur /usr/bin/apt-key.old /usr/bin/apt-key
--- /usr/bin/apt-key.old    2023-09-01 19:46:18.627613000 +0000
+++ /usr/bin/apt-key    2023-09-01 19:40:30.985301000 +0000
@@ -1,6 +1,7 @@
 #!/bin/sh
 
 set -e
+if [ -n "$MY_DEBUG_SLEEP" ]; then sleep 3600; fi
 
 unset GREP_OPTIONS GPGHOMEDIR CURRENTTRAP
 export IFS="$(printf "\n\b")"
@@ -544,6 +545,7 @@
     elif [ -z "$FORCED_KEYRING" ]; then
     merge_all_trusted_keyrings_into_pubring
     if [ -r "${GPGHOMEDIR}/pubring.gpg" ]; then
+            ls -la "${GPGHOMEDIR}"
         cp -a "${GPGHOMEDIR}/pubring.gpg" "${GPGHOMEDIR}/pubring.orig.gpg"
     else
        touch "${GPGHOMEDIR}/pubring.gpg" "${GPGHOMEDIR}/pubring.orig.gpg"

Shows the problem occurring here:

Code:
root@freebsd14:/# MY_DEBUG_SLEEP=Y apt -oAPT::Status-Fd=1 -oDebug::Acquire::gpgv=1 update
root@freebsd14:/# ps auxfww
root       949  0.0  0.1  13404  2964 ?        Ss   15:39   0:00 login [pam]
root       950  0.0  0.1  13368  3152 ?        S    15:39   0:01  \_ -sh
root     25160  0.0  0.2   6036  4400 ?        S    18:11   0:03      \_ /bin/bash
root     26053  0.0  0.5  14720 11140 ?        T    18:25   0:00          \_ apt -oAPT::Status-Fd=1 -oDebug::Acquire::gpgv=1 update
_apt     26056  0.0  0.7  20448 14932 ?        T    18:25   0:00          |   \_ /usr/lib/apt/methods/http
_apt     26058  0.0  0.4  13884  9832 ?        T    18:25   0:00          |   \_ /usr/lib/apt/methods/gpgv
_apt     26059  0.0  0.4  14140  9944 ?        T    18:25   0:00          |       \_ /usr/lib/apt/methods/gpgv
_apt     26060  0.0  0.0   2464  1652 ?        T    18:25   0:00          |           \_ /bin/sh /usr/bin/apt-key --quiet --readonly verify --status-fd 3 /tmp/apt.sig.OO1MDD /tmp/apt.data.FtsGdD
_apt     26061  0.0  0.0   2376  1552 ?        T    18:25   0:00          |               \_ sleep 3600

root@freebsd14:~# su _apt -s /bin/dash
$ id
uid=100(_apt) gid=65534(nogroup) groups=65534(nogroup)

_apt@freebsd14:/$ /bin/sh -x /usr/bin/apt-key --quiet --readonly verify --status-fd 1 /tmp/apt.sig.OO1MDD /tmp/apt.data.FtsGdD
# ...
+ ls -la /tmp/apt-key-gpghome.ptBlRcbxDY
ls: /tmp/apt-key-gpghome.ptBlRcbxDY: Operation not permitted
ls: /tmp/apt-key-gpghome.ptBlRcbxDY/.: Operation not permitted
ls: /tmp/apt-key-gpghome.ptBlRcbxDY/..: Operation not permitted
ls: /tmp/apt-key-gpghome.ptBlRcbxDY/pubring.gpg: Operation not permitted
total 56K
drwx------ 2 _apt root    64 Sep  1 19:41 .
drwxrwxrwt 3 root root   256 Sep  1 19:41 ..
-rw-r--r-- 1 _apt root 56156 Sep  1 19:41 pubring.gpg
+ cp -a /tmp/apt-key-gpghome.ptBlRcbxDY/pubring.gpg /tmp/apt-key-gpghome.ptBlRcbxDY/pubring.orig.gpg
cp: ‘/tmp/apt-key-gpghome.ptBlRcbxDY/pubring.gpg’: Operation not permitted

The above commands work successfully under FreeBSD 13.1.

The issue seems to be with the cp -a, which expands to something like -dR --preserve=mode,ownership,timestamps,context,links,xattr, where some of those are optional. (It seems cp -a is unnecessary in apt-key especially since it is now run as a non-root user, but that is beside the point).

Here is a test showing different behaviour for 2 different --preserve options between FreeBSD 13.1 and 14.0-ALPHA3. This is on a UFS chroot directory. cp -a requires successful mode execution whereas xattr is optional. So it looks like the issue in this thread might just be caused by fgetxattr.

Code:
someguy@freebsd13:~$ strace cp --preserve=mode hello hello2
fgetxattr(3, "system.posix_acl_access", 0x7fffffffdcc0, 132) = -1 EOPNOTSUPP (Operation not supported)
fsetxattr(4, "system.posix_acl_access", "\2\0\0\0\1\0\6\0\377\377\377\377\4\0\4\0\377\377\377\377 \0\4\0\377\377\377\377", 28, 0) = -1 EOPNOTSUPP (Operation not supported)
+++ exited with 0 +++

someguy@freebsd14:~$ strace cp --preserve=mode hello hello2
fgetxattr(3, "system.posix_acl_access", 0x7fffffffbd30, 132) = -1 EPERM (Operation not permitted)
+++ exited with 1 +++

someguy@freebsd13:~$ strace cp --preserve=xattr hello hello2
flistxattr(3, NULL, 0)                  = -1 EOPNOTSUPP (Operation not supported)
+++ exited with 0 +++

someguy@freebsd14:~$ strace cp --preserve=xattr hello hello2
flistxattr(3, NULL, 0)                  = -1 EPERM (Operation not permitted)
+++ exited with 1 +++

Simple ls -l examples:
Code:
someguy@freebsd13:~$ ls -la
total 24K
drwxr-xr-x 2 someguy someguy  512 Sep  1 21:06 .
drwxr-xr-x 3 root    root     512 Sep  1 20:11 ..
-rw------- 1 someguy someguy   91 Sep  1 20:40 .bash_history
-rw-r--r-- 1 someguy someguy  220 Sep  1 20:11 .bash_logout
-rw-r--r-- 1 someguy someguy 3526 Sep  1 20:11 .bashrc
-rw-r--r-- 1 someguy someguy  807 Sep  1 20:11 .profile
-rw-r--r-- 1 someguy someguy    0 Sep  1 20:12 hello
-rw-r--r-- 1 someguy someguy    0 Sep  1 21:12 hello2

someguy@freebsd13:~$ strace ls -la 2>&1 | grep getxattr
getxattr(".", "system.posix_acl_access", NULL, 0) = -1 EOPNOTSUPP (Operation not supported)

someguy@freebsd14:~$ ls -la
ls: .: Operation not permitted
ls: ..: Operation not permitted
ls: .bash_logout: Operation not permitted
ls: .bashrc: Operation not permitted
ls: .profile: Operation not permitted
ls: hello: Operation not permitted
ls: hello2: Operation not permitted
ls: .bash_history: Operation not permitted
total 24K
drwxr-xr-x 2 someguy someguy  512 Sep  1 21:29 .
drwxr-xr-x 3 root    root     512 Sep  1 20:11 ..
-rw------- 1 someguy someguy   64 Sep  1 20:24 .bash_history
-rw-r--r-- 1 someguy someguy  220 Sep  1 20:11 .bash_logout
-rw-r--r-- 1 someguy someguy 3526 Sep  1 20:11 .bashrc
-rw-r--r-- 1 someguy someguy  807 Sep  1 20:11 .profile
-rw-r--r-- 1 someguy someguy    0 Sep  1 20:11 hello
-rw-r--r-- 1 someguy someguy    0 Sep  1 21:14 hello2

someguy@freebsd14:~$ strace ls -la 2>&1 | grep getxattr
getxattr(".", "system.posix_acl_access", NULL, 0) = -1 EPERM (Operation not permitted)
getxattr("..", "system.posix_acl_access", NULL, 0) = -1 EPERM (Operation not permitted)
getxattr(".bash_logout", "system.posix_acl_access", NULL, 0) = -1 EPERM (Operation not permitted)
getxattr(".bashrc", "system.posix_acl_access", NULL, 0) = -1 EPERM (Operation not permitted)
getxattr(".profile", "system.posix_acl_access", NULL, 0) = -1 EPERM (Operation not permitted)
getxattr("hello", "system.posix_acl_access", NULL, 0) = -1 EPERM (Operation not permitted)
getxattr("hello2", "system.posix_acl_access", NULL, 0) = -1 EPERM (Operation not permitted)
getxattr(".bash_history", "system.posix_acl_access", NULL, 0) = -1 EPERM (Operation not permitted)

Edit: It looks like this or something similar is already a known issue on the mailing lists, but I'm not 100% sure if this context fully matches so I filed a bug report here.
 
"I upgraded to FreeBSD 15 and tested the script again, but this time encountered another issue. Installing gnupg didn't resolve it either."


Code:
root@ykla:/# apt update
Hit:1 http://mirrors.ustc.edu.cn/debian stable InRelease
Hit:2 http://mirrors.ustc.edu.cn/debian stable-updates InRelease
Err:1 http://mirrors.ustc.edu.cn/debian stable InRelease
  Couldn't execute /usr/bin/apt-key to check /var/lib/apt/lists/mirrors.ustc.edu.cn_debian_dists_stable_InRelease
Err:2 http://mirrors.ustc.edu.cn/debian stable-updates InRelease
  Couldn't execute /usr/bin/apt-key to check /var/lib/apt/lists/mirrors.ustc.edu.cn_debian_dists_stable-updates_InRelease
Hit:3 http://mirrors.ustc.edu.cn/debian-security stable-security InRelease
Err:3 http://mirrors.ustc.edu.cn/debian-security stable-security InRelease
  Couldn't execute /usr/bin/apt-key to check /var/lib/apt/lists/mirrors.ustc.edu.cn_debian-security_dists_stable-security_InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://mirrors.ustc.edu.cn/debian stable InRelease: Couldn't execute /usr/bin/apt-key to check /var/lib/apt/lists/mirrors.ustc.edu.cn_debian_dists_stable_InRelease
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://mirrors.ustc.edu.cn/debian stable-updates InRelease: Couldn't execute /usr/bin/apt-key to check /var/lib/apt/lists/mirrors.ustc.edu.cn_debian_dists_stable-updates_InRelease
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://mirrors.ustc.edu.cn/debian-security stable-security InRelease: Couldn't execute /usr/bin/apt-key to check /var/lib/apt/lists/mirrors.ustc.edu.cn_debian-security_dists_stable-security_InRelease
W: Failed to fetch http://mirrors.ustc.edu.cn/debian/dists/stable/InRelease  Couldn't execute /usr/bin/apt-key to check /var/lib/apt/lists/mirrors.ustc.edu.cn_debian_dists_stable_InRelease
W: Failed to fetch http://mirrors.ustc.edu.cn/debian/dists/stable-updates/InRelease  Couldn't execute /usr/bin/apt-key to check /var/lib/apt/lists/mirrors.ustc.edu.cn_debian_dists_stable-updates_InRelease
W: Failed to fetch http://mirrors.ustc.edu.cn/debian-security/dists/stable-security/InRelease  Couldn't execute /usr/bin/apt-key to check /var/lib/apt/lists/mirrors.ustc.edu.cn_debian-security_dists_stable-security_InRelease
W: Some index files failed to download. They have been ignored, or old ones used instead.
root@ykla:/#

If you switch to another mirror site, the issue still persists.

Code:
root@ykla:/# apt update
Get:1 https://mirrors.tuna.tsinghua.edu.cn/debian bookworm InRelease [151 kB]      
Err:1 https://mirrors.tuna.tsinghua.edu.cn/debian bookworm InRelease               
  Couldn't execute /usr/bin/apt-key to check /var/lib/apt/lists/partial/mirrors.tuna.tsinghua.edu.cn_debian_dists_bookworm_InRelease
Get:2 https://mirrors.tuna.tsinghua.edu.cn/debian bookworm-updates InRelease [52.1 kB]
Err:2 https://mirrors.tuna.tsinghua.edu.cn/debian bookworm-updates InRelease             
  Couldn't execute /usr/bin/apt-key to check /var/lib/apt/lists/partial/mirrors.tuna.tsinghua.edu.cn_debian_dists_bookworm-updates_InRelease
Get:3 https://mirrors.tuna.tsinghua.edu.cn/debian bookworm-backports InRelease [56.3 kB]
Err:3 https://mirrors.tuna.tsinghua.edu.cn/debian bookworm-backports InRelease           
  Couldn't execute /usr/bin/apt-key to check /var/lib/apt/lists/partial/mirrors.tuna.tsinghua.edu.cn_debian_dists_bookworm-backports_InRelease
Get:4 https://security.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Err:4 https://security.debian.org/debian-security bookworm-security InRelease
  Couldn't execute /usr/bin/apt-key to check /var/lib/apt/lists/partial/security.debian.org_debian-security_dists_bookworm-security_InRelease
Reading package lists... Done
W: GPG error: https://mirrors.tuna.tsinghua.edu.cn/debian bookworm InRelease: Couldn't execute /usr/bin/apt-key to check /var/lib/apt/lists/partial/mirrors.tuna.tsinghua.edu.cn_debian_dists_bookworm_InRelease
E: The repository 'https://mirrors.tuna.tsinghua.edu.cn/debian bookworm InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: GPG error: https://mirrors.tuna.tsinghua.edu.cn/debian bookworm-updates InRelease: Couldn't execute /usr/bin/apt-key to check /var/lib/apt/lists/partial/mirrors.tuna.tsinghua.edu.cn_debian_dists_bookworm-updates_InRelease
E: The repository 'https://mirrors.tuna.tsinghua.edu.cn/debian bookworm-updates InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: GPG error: https://mirrors.tuna.tsinghua.edu.cn/debian bookworm-backports InRelease: Couldn't execute /usr/bin/apt-key to check /var/lib/apt/lists/partial/mirrors.tuna.tsinghua.edu.cn_debian_dists_bookworm-backports_InRelease
E: The repository 'https://mirrors.tuna.tsinghua.edu.cn/debian bookworm-backports InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: GPG error: https://security.debian.org/debian-security bookworm-security InRelease: Couldn't execute /usr/bin/apt-key to check /var/lib/apt/lists/partial/security.debian.org_debian-security_dists_bookworm-security_InRelease
E: The repository 'https://security.debian.org/debian-security bookworm-security InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
root@ykla:/#


1693875872730.png
 
After some digging around, it looks like this issue *might* be due to a difference between 13.x and 14.x in how the Linux binary emulation handles file attribute syscalls for non-root users inside chroots, specifically fgetxattr returning EPERM which prevents cp --preserve=mode from executing, and also causes errors in typical ls -l output.

It could also be something with my setup, but I am using clean VM installs. In any case, I'll check for bug reports and mailing list posts and file one if not already known.

---

Continuing from the debootstrap example above and applying this patch inside the chroot:

Code:
root@freebsd14:/# diff -Naur /usr/bin/apt-key.old /usr/bin/apt-key
--- /usr/bin/apt-key.old    2023-09-01 19:46:18.627613000 +0000
+++ /usr/bin/apt-key    2023-09-01 19:40:30.985301000 +0000
@@ -1,6 +1,7 @@
 #!/bin/sh
 
 set -e
+if [ -n "$MY_DEBUG_SLEEP" ]; then sleep 3600; fi
 
 unset GREP_OPTIONS GPGHOMEDIR CURRENTTRAP
 export IFS="$(printf "\n\b")"
@@ -544,6 +545,7 @@
     elif [ -z "$FORCED_KEYRING" ]; then
     merge_all_trusted_keyrings_into_pubring
     if [ -r "${GPGHOMEDIR}/pubring.gpg" ]; then
+            ls -la "${GPGHOMEDIR}"
         cp -a "${GPGHOMEDIR}/pubring.gpg" "${GPGHOMEDIR}/pubring.orig.gpg"
     else
        touch "${GPGHOMEDIR}/pubring.gpg" "${GPGHOMEDIR}/pubring.orig.gpg"

Shows the problem occurring here:

Code:
root@freebsd14:/# MY_DEBUG_SLEEP=Y apt -oAPT::Status-Fd=1 -oDebug::Acquire::gpgv=1 update
root@freebsd14:/# ps auxfww
root       949  0.0  0.1  13404  2964 ?        Ss   15:39   0:00 login [pam]
root       950  0.0  0.1  13368  3152 ?        S    15:39   0:01  \_ -sh
root     25160  0.0  0.2   6036  4400 ?        S    18:11   0:03      \_ /bin/bash
root     26053  0.0  0.5  14720 11140 ?        T    18:25   0:00          \_ apt -oAPT::Status-Fd=1 -oDebug::Acquire::gpgv=1 update
_apt     26056  0.0  0.7  20448 14932 ?        T    18:25   0:00          |   \_ /usr/lib/apt/methods/http
_apt     26058  0.0  0.4  13884  9832 ?        T    18:25   0:00          |   \_ /usr/lib/apt/methods/gpgv
_apt     26059  0.0  0.4  14140  9944 ?        T    18:25   0:00          |       \_ /usr/lib/apt/methods/gpgv
_apt     26060  0.0  0.0   2464  1652 ?        T    18:25   0:00          |           \_ /bin/sh /usr/bin/apt-key --quiet --readonly verify --status-fd 3 /tmp/apt.sig.OO1MDD /tmp/apt.data.FtsGdD
_apt     26061  0.0  0.0   2376  1552 ?        T    18:25   0:00          |               \_ sleep 3600

root@freebsd14:~# su _apt -s /bin/dash
$ id
uid=100(_apt) gid=65534(nogroup) groups=65534(nogroup)

_apt@freebsd14:/$ /bin/sh -x /usr/bin/apt-key --quiet --readonly verify --status-fd 1 /tmp/apt.sig.OO1MDD /tmp/apt.data.FtsGdD
# ...
+ ls -la /tmp/apt-key-gpghome.ptBlRcbxDY
ls: /tmp/apt-key-gpghome.ptBlRcbxDY: Operation not permitted
ls: /tmp/apt-key-gpghome.ptBlRcbxDY/.: Operation not permitted
ls: /tmp/apt-key-gpghome.ptBlRcbxDY/..: Operation not permitted
ls: /tmp/apt-key-gpghome.ptBlRcbxDY/pubring.gpg: Operation not permitted
total 56K
drwx------ 2 _apt root    64 Sep  1 19:41 .
drwxrwxrwt 3 root root   256 Sep  1 19:41 ..
-rw-r--r-- 1 _apt root 56156 Sep  1 19:41 pubring.gpg
+ cp -a /tmp/apt-key-gpghome.ptBlRcbxDY/pubring.gpg /tmp/apt-key-gpghome.ptBlRcbxDY/pubring.orig.gpg
cp: ‘/tmp/apt-key-gpghome.ptBlRcbxDY/pubring.gpg’: Operation not permitted

The above commands work successfully under FreeBSD 13.1.

The issue seems to be with the cp -a, which expands to something like -dR --preserve=mode,ownership,timestamps,context,links,xattr, where some of those are optional. (It seems cp -a is unnecessary in apt-key especially since it is now run as a non-root user, but that is beside the point).

Here is a test showing different behaviour for 2 different --preserve options between FreeBSD 13.1 and 14.0-ALPHA3. This is on a UFS chroot directory. cp -a requires successful mode execution whereas xattr is optional. So it looks like the issue in this thread might just be caused by fgetxattr.

Code:
someguy@freebsd13:~$ strace cp --preserve=mode hello hello2
fgetxattr(3, "system.posix_acl_access", 0x7fffffffdcc0, 132) = -1 EOPNOTSUPP (Operation not supported)
fsetxattr(4, "system.posix_acl_access", "\2\0\0\0\1\0\6\0\377\377\377\377\4\0\4\0\377\377\377\377 \0\4\0\377\377\377\377", 28, 0) = -1 EOPNOTSUPP (Operation not supported)
+++ exited with 0 +++

someguy@freebsd14:~$ strace cp --preserve=mode hello hello2
fgetxattr(3, "system.posix_acl_access", 0x7fffffffbd30, 132) = -1 EPERM (Operation not permitted)
+++ exited with 1 +++

someguy@freebsd13:~$ strace cp --preserve=xattr hello hello2
flistxattr(3, NULL, 0)                  = -1 EOPNOTSUPP (Operation not supported)
+++ exited with 0 +++

someguy@freebsd14:~$ strace cp --preserve=xattr hello hello2
flistxattr(3, NULL, 0)                  = -1 EPERM (Operation not permitted)
+++ exited with 1 +++

Simple ls -l examples:
Code:
someguy@freebsd13:~$ ls -la
total 24K
drwxr-xr-x 2 someguy someguy  512 Sep  1 21:06 .
drwxr-xr-x 3 root    root     512 Sep  1 20:11 ..
-rw------- 1 someguy someguy   91 Sep  1 20:40 .bash_history
-rw-r--r-- 1 someguy someguy  220 Sep  1 20:11 .bash_logout
-rw-r--r-- 1 someguy someguy 3526 Sep  1 20:11 .bashrc
-rw-r--r-- 1 someguy someguy  807 Sep  1 20:11 .profile
-rw-r--r-- 1 someguy someguy    0 Sep  1 20:12 hello
-rw-r--r-- 1 someguy someguy    0 Sep  1 21:12 hello2

someguy@freebsd13:~$ strace ls -la 2>&1 | grep getxattr
getxattr(".", "system.posix_acl_access", NULL, 0) = -1 EOPNOTSUPP (Operation not supported)

someguy@freebsd14:~$ ls -la
ls: .: Operation not permitted
ls: ..: Operation not permitted
ls: .bash_logout: Operation not permitted
ls: .bashrc: Operation not permitted
ls: .profile: Operation not permitted
ls: hello: Operation not permitted
ls: hello2: Operation not permitted
ls: .bash_history: Operation not permitted
total 24K
drwxr-xr-x 2 someguy someguy  512 Sep  1 21:29 .
drwxr-xr-x 3 root    root     512 Sep  1 20:11 ..
-rw------- 1 someguy someguy   64 Sep  1 20:24 .bash_history
-rw-r--r-- 1 someguy someguy  220 Sep  1 20:11 .bash_logout
-rw-r--r-- 1 someguy someguy 3526 Sep  1 20:11 .bashrc
-rw-r--r-- 1 someguy someguy  807 Sep  1 20:11 .profile
-rw-r--r-- 1 someguy someguy    0 Sep  1 20:11 hello
-rw-r--r-- 1 someguy someguy    0 Sep  1 21:14 hello2

someguy@freebsd14:~$ strace ls -la 2>&1 | grep getxattr
getxattr(".", "system.posix_acl_access", NULL, 0) = -1 EPERM (Operation not permitted)
getxattr("..", "system.posix_acl_access", NULL, 0) = -1 EPERM (Operation not permitted)
getxattr(".bash_logout", "system.posix_acl_access", NULL, 0) = -1 EPERM (Operation not permitted)
getxattr(".bashrc", "system.posix_acl_access", NULL, 0) = -1 EPERM (Operation not permitted)
getxattr(".profile", "system.posix_acl_access", NULL, 0) = -1 EPERM (Operation not permitted)
getxattr("hello", "system.posix_acl_access", NULL, 0) = -1 EPERM (Operation not permitted)
getxattr("hello2", "system.posix_acl_access", NULL, 0) = -1 EPERM (Operation not permitted)
getxattr(".bash_history", "system.posix_acl_access", NULL, 0) = -1 EPERM (Operation not permitted)

Edit: It looks like this or something similar is already a known issue on the mailing lists, but I'm not 100% sure if this context fully matches so I filed a bug report here.
It seems unrelated to the fix mentioned by the developers in the bug. I tested it on FreeBSD 15, and it still doesn't work, and I've always been using the root user.
 
it looks like this issue *might* be due to a difference between 13.x and 14.x in how the Linux binary emulation handles file attribute syscalls for non-root users inside chroots, specifically fgetxattr returning EPERM which prevents cp --preserve=mode from executing, and also causes errors in typical ls -l output.
It's not exactly a difference but ... until recently, the Linux xattr syscalls were just unimplemented, so they did nothing and always returned ENOTSUP. As not all filesystems can store extended attributes, this isn't considered fatal by any tools.

Now, after Linux xattr support was added, I ran into this very issue that it broke a few tools. I'll try to summarize what I learned in the analysis and discussion on the mailing list. Extended attributes have namespaces, and access to the "system" namespace is typically restricted, both for non-root users and inside jails even for the superuser. On Linux, it's restricted for non-root as well, although according to dchagin, it's the filesystems implementing the policies (while in FreeBSD, this is done in a central place). When FreeBSD doesn't allow some access, the error code is EPERM, as you would probably expect.

Now, the strage thing is, that's different on Linux. When you try to write/delete attributes in "system" namespace, or list all attributes (which would include "system"), the error code is ENOTSUP. Trying to read one will give you ENOATTR. I can only guess that's a result of letting the filesystems implement the access policies. The consequence is, GNU/Linux tools will consider EPERM a fatal error condition, because they don't expect to get it when they're just not allowed to do the access.

The fix is to add some error-code mapping trickery to the Linux syscall wrappers. This was indeed already committed to 15-CURRENT (and will be merged to stable/14), but a little piece was accidentally missing, the mapping of the error code for attempts to list attributes wasn't correct. Now, that's exactly what cp from GNU coreutils attempts to do when called with cp -p.

A patch for this is available here: https://people.freebsd.org/~dchagin/lxattr.patch -- I just finished thoroughly testing it: https://lists.freebsd.org/archives/freebsd-current/2023-September/004582.html

So I assume it will be committed soon, and of course also merged to stable/14.
 
Then update your 15-CURRENT again. The (hopefully final) fix for linux xattr syscalls was just committed today.
Unfortunately, the malfunction still exists.

FreeBSD ykla 15.0-CURRENT FreeBSD 15.0-CURRENT amd64 1500000 #0 main-165234d55: Wed Sep 6 08:01:29 CST 2023 root@ykla:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64
 
Yes, but are the forums really a good place for debugging alpha or beta versions of a yet to be RELEASE version of FreeBSD?
Apart from a good (or most appropriate) place it *is* 14-STABLE; therefore it falls within the guidelines of the forum (the formalistic approach).

A -STABLE version may be seen as a nice bridge between a "dull" binary (non "built-by-yourself") supported FreeBSD -RELEASE version and a less "dull" build of a -RELEASE version by oneself versus the very fluid build and run environment of all that is -CURRENT. -CURRENT being only accompanied by the very different eco system of the mailing lists—as opposed to a forum setting. A decision has to be made where -STABLE will be supported. IMO, the present division is a good, defensible and a clear one.

Were one to decide to not support -STABLE on this forum when there is not yet an accompanying -RELEASE present that would perhaps represent a clear division as well but, IMO, a less defensible and logical one. It would be somewhat tantamount to: "We will not let us help each other on this forum and we will not discuss building/compiling/running 'exiting' and challenging -STABLE stuff here until there has been issued an accompanying -RELEASE version. For this small period of time, you will be 'banned' to the mailing lists!" Probably also not the most logical environment for creating the next generation of enthousiastic supporters, documenters and developers of FreeBSD. Lastly, it would present an awkward contradiction where this particular kind of -STABLE is officially supported by FreeBSD (see also my earlier message about the FreeBSD support model), but, notably, not on this forum.
 
Yes, but are the forums really a good place for debugging alpha or beta versions of a yet to be RELEASE version of FreeBSD?
FreeBSD 14.0-BETA1 will be on the releng/14.0 branch, so we shouldn't support that one either? Isn't the whole point of alpha and beta releases to widen the audience in order to catch the bugs developers haven't found yet? We can't 'solve' any bugs here on the forums, even for -RELEASE versions. So you'll get directed to bugzilla and/or the mailing lists in those cases. We can help 'identify' and narrow down potential bugs though.
 
A -STABLE version may be seen as a nice bridge between a "dull" binary (non "built-by-yourself") supported FreeBSD -RELEASE version and a less "dull" build of a -RELEASE version by oneself versus the very fluid build and run environment of all that is -CURRENT. -CURRENT being only accompanied by the very different eco system of the mailing lists—as opposed to a forum setting. A decision has to be made where -STABLE will be supported. IMO, the present division is a good, defensible and a clear one.

Isn't the whole point of alpha and beta releases to widen the audience in order to catch the bugs developers haven't found yet? We can't 'solve' any bugs here on the forums, even for -RELEASE versions. So you'll get directed to bugzilla and/or the mailing lists in those cases. We can help 'identify' and narrow down potential bugs though.

Thankyou both for clarification and detail.

This old dog missed some years due to illness, and last ran -STABLE at 9.3, last built Oct '15 when stable@ was still a very active list as well as current@; a quick illustration being the sizes of monthly archives then compared to the contemporary now.

So yes, the present reality is quite different; I did speak out of turn, and appreciate not being spanked too hard! <&^}=
 
I'm sorry, I'm not very familiar with the debugging tools you mentioned. I've only been writing some simple Linux compatibility layer build scripts with the help of others. Could you please elaborate on what I should do?
Then it's IMHO unlikely that this is still related to the xattr support ... but not impossible of course.

Can you analyze which command exactly fails and then collect a truss(1) log from that?
 
ykla unfortunately, I can't give you a recipe ... from your log posted above, it's either /usr/bin/apt-key failing, or apt already fails to invoke it for some reason. What will be needed for analysis is to narrow down the problem to the exact command that fails, this may involve analyzing apt source code and/or scripts ...

Once you know exactly the failing command, it's kind of simple: run it with truss(1) and redirect stderr to some log file. For a Linux binary, this must be done from FreeBSD calling the command with /compat/linux (or whatever your linuxulator base is) prepended.

Oh and if you still don't understand a word, maybe someone else can help here 😉 I'm not sure whether I could reproduce it, but at least not right now, my -CURRENT machines are busy ....
 
It seems unrelated to the fix mentioned by the developers in the bug. I tested it on FreeBSD 15, and it still doesn't work, and I've always been using the root user.

When you run apt update as root, apt-file is executed as the _apt user. See my previous comment for more details on how to debug whats going on.

The issue seems fixed in latest FreeBSD-15.0-CURRENT-amd64-20230907 build (see this comment in the bug report). Pending merge into 14.0 tree.
 
Yes, but are the forums really a good place for debugging alpha or beta versions of a yet to be RELEASE version of FreeBSD?
Isn't the whole point of alpha and beta releases to widen the audience in order to catch the bugs developers haven't found yet? We can't 'solve' any bugs here on the forums, even for -RELEASE versions. So you'll get directed to bugzilla and/or the mailing lists in those cases. We can help 'identify' and narrow down potential bugs though.

A solution could be to have a new forum section for Alpha/Beta threads. Any thread like this one could then be moved there by moderators, to keep the rest of the forum nice and tidy with long-lasting information for released versions. These sort-of bug report threads can be noisy, and once the underlying issue is fixed, they still hang around on the forums with outdated irrelevant information. So having them in their own clearly marked section shows to future readers that this was a discussion for an ALPHA/BETA release and they can decide how relevant it still is.

Using this thread as an example, I ran into the original issue using FreeBSD 14.0-ALPHA3, then searched Google for the exact error and only found this thread, so continued the discussion here.

Was it a bug, was it new behaviour, was it a known issue?... sometimes it is only possible to find out after asking and investigating more. In this case, it turned out to be a real bug (debatable!), and zirias@ provided a helpful and easy to understand summary of the original mailing list thread which I wasn't able to follow. I agree the real place to centralize all discussion for specific bugs is on bugzilla. It can be difficult to jump into existing discussions on mailing lists if you are not already subscribed and set up for it, and it might just add a lot of noise depending on the situation. These forums and bugzilla are much more accessible, and the forums offer more casual discussion than the dry details of a bug report. Just my 2c anyway 😉
 
When you run apt update as root, apt-file is executed as the _apt user. See my previous comment for more details on how to debug whats going on.

The issue seems fixed in latest FreeBSD-15.0-CURRENT-amd64-20230907 build (see this comment in the bug report). Pending merge into 14.0 tree.
But it seems like I've encountered a new problem, stuck below.


root@ykla:~ # uname -a
FreeBSD ykla 15.0-CURRENT FreeBSD 15.0-CURRENT amd64 1500000 #0 main-9c80d66ec: Sat Sep 9 14:41:10 CST 2023 root@ykla:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64
--------------------------------------------------------------------------------
W: Cannot check Release signature; keyring file not available /usr/local/share/keyrings/debian-archive-keyring.gpg
I: Validating Packages
I: Resolving dependencies of required packages...
 
But it seems like I've encountered a new problem, stuck below.


root@ykla:~ # uname -a
FreeBSD ykla 15.0-CURRENT FreeBSD 15.0-CURRENT amd64 1500000 #0 main-9c80d66ec: Sat Sep 9 14:41:10 CST 2023 root@ykla:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64
--------------------------------------------------------------------------------
W: Cannot check Release signature; keyring file not available /usr/local/share/keyrings/debian-archive-keyring.gpg
I: Validating Packages
I: Resolving dependencies of required packages...

This issue was already present on 13.1:

Code:
# uname -a
FreeBSD freebsd13 13.2-RELEASE-p2 FreeBSD 13.2-RELEASE-p2 GENERIC amd64

# pkg install sysutils/debootstrap

# ls /usr/local/share/keyrings/
ubuntu-archive-keyring.gpg              ubuntu-keyring-2012-cdimage.gpg
ubuntu-archive-removed-keys.gpg         ubuntu-keyring-2012-cloud-archive.gpg
ubuntu-cloud-keyring.gpg                ubuntu-keyring-2016-dbgsym.gpg
ubuntu-cloud-removed-keys.gpg           ubuntu-keyring-2018-archive.gpg
ubuntu-cloudimage-keyring.gpg           ubuntu-keyring-2020-oem.gpg
ubuntu-cloudimage-removed-keys.gpg      ubuntu-master-keyring.gpg
ubuntu-dbgsym-keyring.gpg               ubuntu-oem-keyring.gpg
ubuntu-dbgsym-removed-keys.gpg          ubuntu-oem-removed-keys.gpg
ubuntu-keyring-2008-oem.gpg

# debootstrap bullseye debian_build http://deb.debian.org/debian
W: Cannot check Release signature; keyring file not available /usr/local/share/keyrings/debian-archive-keyring.gpg

As the warning states, installing the sysutils/debootstrap package doesn't install the Debian repository GPG keys by default, only the security/ubuntu-keyring. I'm not sure why the Debian keys are not similarly packaged, maybe no-one has gotten around to it yet?
 
jwmullally thanks a lot for doing the issue analysis in this case. So, the issues with the initial implementation of Linux xattr syscalls are indeed all solved now (on main, still pending MFC). That's in line with my findings running into the same issues while trying to build software targeting Linux (where several "install" steps failed, caused by a variety of tools like "install" and "cp" from GNU coreutils, as well as some stuff using a python built for Linux .... and they all had issues with xattr syscalls which are all fixed now as well).
 
Back
Top