Solved Jails, permissions issues

dervishe · Apr 18, 2023

Hello,
i'm using zfs dataset with jails and i meet some troubles with permissions issue:
One of my jails is used for mariadb and is built on 3 datasets automatically mounted at boot:

Bash:

# mount | grep db
zroot/jails/db on /jails/db (zfs, local, noatime, nfsv4acls)
zroot/jails/db/data on /jails/db/var/db/mysql/data (zfs, local, noatime, nfsv4acls)
zroot/jails/db/log on /jails/db/var/db/mysql/log (zfs, local, noatime, nfsv4acls)
devfs on /jails/db/dev (devfs)

The trouble is that within the jail, the user mysql can't write in the directories:
- /jails/db/var/db/mysql/data
- /jails/db/var/db/mysql/log

The permissions for those directories are:

Bash:

# ls -l /var/db/mysql
drwxr-xr-x  2 mysql  mysql  2 Apr 13 12:28 data/
drwxr-xr-x  2 mysql  mysql  2 Apr  5 10:31 log/

I put here the result of getfacl in the jail:

Bash:

dervishe@db:/var/db# getfacl -v /var/db/mysql/data/
# file: /var/db/mysql/data/
# owner: mysql
# group: mysql
            owner@:read_data/write_data/execute/append_data/read_attributes/write_attributes/read_xattr/write_xattr/read_acl/write_acl/write_owner/synchronize::allow
            group@:read_data/execute/read_attributes/read_xattr/read_acl/synchronize::allow
         everyone@:read_data/execute/read_attributes/read_xattr/read_acl/synchronize::allow

dervishe@db:/var/db# getfacl -v /var/db/mysql/log
# file: /var/db/mysql/log
# owner: mysql
# group: mysql
            owner@:read_data/write_data/execute/append_data/read_attributes/write_attributes/read_xattr/write_xattr/read_acl/write_acl/write_owner/synchronize::allow
            group@:read_data/execute/read_attributes/read_xattr/read_acl/synchronize::allow
         everyone@:read_data/execute/read_attributes/read_xattr/read_acl/synchronize::allow

The config for the jail is:

Bash:

db {
    host.hostname = db;
    /*Just network stuff here*/
}

My rc.conf in the jail:

Code:

db:/var/db/mysql# cat /etc/rc.conf
mysql_enable="YES"
mysql_dbdir=/var/db/mysql/data

The /etc/sysctl.conf in the Host:

Bash:

# cat /etc/sysctl.conf

#security.bsd.see_other_uids=0
security.bsd.see_other_uids=0
security.bsd.see_other_gids=0
security.bsd.see_jail_proc=0
security.bsd.unprivileged_read_msgbuf=0
security.bsd.unprivileged_proc_debug=0
kern.randompid=1
vfs.zfs.min_auto_ashift=12
hw.spec_store_bypass_disable=2
hw.ibrs_disable=0

i'm running: 13.1-RELEASE-p5 FreeBSD 13.1-RELEASE-p5 releng/13.1-n250174-753d65a19a55

I'm missing something but i can't see what it is...
Any hints ?
Thanks a lot for your help

Edit: just added the getfacl results

SirDice · Apr 18, 2023

What are the permissions on /var/db/mysql itself? ls -ld /var/db/mysql

dervishe · Apr 18, 2023

Thx for your reply

The permissions are: (ls and getfacl)

Code:

# ls -ld
drwxr-xr-x  4 mysql  mysql  4 Oct 27  2021 /var/db/mysql

# getfacl -v /var/db/mysql
# file: /var/db/mysql
# owner: mysql
# group: mysql
            owner@:read_data/write_data/execute/append_data/read_attributes/write_attributes/read_xattr/write_xattr/read_acl/write_acl/write_owner/synchronize::allow
            group@:read_data/execute/read_attributes/read_xattr/read_acl/synchronize::allow
         everyone@:read_data/execute/read_attributes/read_xattr/read_acl/synchronize::allow

SirDice · Apr 18, 2023

What's the exact message you're getting when you try to start MySQL/MariaDB?

dervishe · Apr 18, 2023

Bash:

Installing MariaDB/MySQL system tables in '/var/db/mysql/data' ...
/usr/local/libexec/mysqld: Can't create file '/var/db/mysql/log/mysqld.err' (errno: 13 "Permission denied")
2023-04-18 17:25:07 0 [ERROR] mysqld: File '/var/db/mysql/log/mysql-bin.index' not found (Errcode: 13 "Permission denied")
2023-04-18 17:25:07 0 [ERROR] Aborting

SirDice · Apr 18, 2023

It has no problems with the data directory. Any existing files in that log directory that may have the wrong permissions and/or ownership?

dervishe · Apr 18, 2023

In fact there are both existing with the same permissions (cf my first message) and log is empty...
I don't know what to do...

SirDice · Apr 18, 2023

From the host, can you do a zfs get all zroot/jails/db/log?

dervishe · Apr 18, 2023

Yep:

Bash:

# zfs get all zroot/jails/db/log
NAME                PROPERTY              VALUE                       SOURCE
zroot/jails/db/log  type                  filesystem                  -
zroot/jails/db/log  creation              Tue Oct 26 21:34 2021       -
zroot/jails/db/log  used                  140K                        -
zroot/jails/db/log  available             40.1T                       -
zroot/jails/db/log  referenced            140K                        -
zroot/jails/db/log  compressratio         1.00x                       -
zroot/jails/db/log  mounted               yes                         -
zroot/jails/db/log  quota                 none                        default
zroot/jails/db/log  reservation           none                        default
zroot/jails/db/log  recordsize            128K                        default
zroot/jails/db/log  mountpoint            /jails/db/var/db/mysql/log  local
zroot/jails/db/log  sharenfs              off                         default
zroot/jails/db/log  checksum              on                          default
zroot/jails/db/log  compression           off                         local
zroot/jails/db/log  atime                 off                         local
zroot/jails/db/log  devices               on                          default
zroot/jails/db/log  exec                  on                          default
zroot/jails/db/log  setuid                on                          default
zroot/jails/db/log  readonly              off                         inherited from zroot
zroot/jails/db/log  jailed                off                         default
zroot/jails/db/log  snapdir               hidden                      default
zroot/jails/db/log  aclmode               discard                     default
zroot/jails/db/log  aclinherit            restricted                  default
zroot/jails/db/log  createtxg             15320                       -
zroot/jails/db/log  canmount              on                          default
zroot/jails/db/log  xattr                 on                          default
zroot/jails/db/log  copies                1                           default
zroot/jails/db/log  version               5                           -
zroot/jails/db/log  utf8only              off                         -
zroot/jails/db/log  normalization         none                        -
zroot/jails/db/log  casesensitivity       sensitive                   -
zroot/jails/db/log  vscan                 off                         default
zroot/jails/db/log  nbmand                off                         default
zroot/jails/db/log  sharesmb              off                         default
zroot/jails/db/log  refquota              none                        default
zroot/jails/db/log  refreservation        none                        default
zroot/jails/db/log  guid                  4013802107587588649         -
zroot/jails/db/log  primarycache          metadata                    local
zroot/jails/db/log  secondarycache        all                         default
zroot/jails/db/log  usedbysnapshots       0B                          -
zroot/jails/db/log  usedbydataset         140K                        -
zroot/jails/db/log  usedbychildren        0B                          -
zroot/jails/db/log  usedbyrefreservation  0B                          -
zroot/jails/db/log  logbias               latency                     default
zroot/jails/db/log  objsetid              4431                        -
zroot/jails/db/log  dedup                 off                         default
zroot/jails/db/log  mlslabel              none                        default
zroot/jails/db/log  sync                  standard                    default
zroot/jails/db/log  dnodesize             legacy                      default
zroot/jails/db/log  refcompressratio      1.00x                       -
zroot/jails/db/log  written               140K                        -
zroot/jails/db/log  logicalused           42.5K                       -
zroot/jails/db/log  logicalreferenced     42.5K                       -
zroot/jails/db/log  volmode               default                     default
zroot/jails/db/log  filesystem_limit      none                        default
zroot/jails/db/log  snapshot_limit        none                        default
zroot/jails/db/log  filesystem_count      none                        default
zroot/jails/db/log  snapshot_count        none                        default
zroot/jails/db/log  snapdev               hidden                      default
zroot/jails/db/log  acltype               nfsv4                       default
zroot/jails/db/log  context               none                        default
zroot/jails/db/log  fscontext             none                        default
zroot/jails/db/log  defcontext            none                        default
zroot/jails/db/log  rootcontext           none                        default
zroot/jails/db/log  relatime              off                         default
zroot/jails/db/log  redundant_metadata    all                         default
zroot/jails/db/log  overlay               on                          default
zroot/jails/db/log  encryption            off                         default
zroot/jails/db/log  keylocation           none                        default
zroot/jails/db/log  keyformat             none                        default
zroot/jails/db/log  pbkdf2iters           0                           default
zroot/jails/db/log  special_small_blocks  0                           default

SirDice · Apr 18, 2023

I was particularly interested in this one as it's often overlooked:

Code:

zroot/jails/db/log  readonly              off                         inherited from zroot

But it's correct, so that can't be the problem.

Emrion · Apr 19, 2023

Perhaps a stupid idea, but I would compare the numerical id of the owner and the user.
I mean: ls -ln /jails/db/var/db/mysql and cat /jails/db/etc/passwd|grep mysql.

dervishe · Apr 19, 2023

Tried it, thx for the idea

But all is good: user mysql (88:88) and the same id/gid for the directories...

SirDice · Apr 19, 2023

I'm kind of running out of ideas. Have you tried just destroying that log dataset and recreating it? Maybe something got whacked when it was created and we're all just overlooking something obvious.

dervishe · Apr 20, 2023

i did a test:
i stopped the jail and unmount the 2 dataset:
* zroot/jails/db/data,
* zroot/jails/db/log
then resume the jail...
and i got the same error.
I have no idea of what is going on here...

freebuser · Apr 20, 2023

What happens if you try to create a file as user mysql

dervishe · Apr 21, 2023

It failed miserably, but i think i got a clue

ltning · Apr 21, 2023

Copying my response from the Fediverse here.
TL;DR: I suspect the problem is traversal issues on /var or /var/db; if the modes of those directories do not allow the mysql user/group to traverse (+x), then that would explain everything. The main hint being the "file not found" error in the log (which is a result of "permission denied").

@dervishe@mastodon.sdf.org What are the permissions on /var and /var/db? The one error from mysql about not creating the log file is not very interesting, but the other one - "file not found" - is *very* interesting: It could/would mean it is not able to traverse the directories above the mysql directory..

@ltning@pleroma.anduin.net Thanks for your reply

All the permissions are described in the forum post. for the file not found it is normal because it's a fresh install without all the files yet created

@dervishe@mastodon.sdf.org I can't find any mention of permissions on /var and /var/db in the post, and those are the ones I mean. Also the "file not found" is absolutely interesting precisely because it says "not found *because of* permission denied". It means it cannot determine the existence of the file because it cannot traverse the parent directories. Remember that traversal of directories needs +x permission - if that's missing on /var/db (if it's mode 750 or something, for example) then that would explain everything you're seeing.

Alain De Vos · Apr 21, 2023

/var/db/mysql should belong to the mysql user.
And all contents of this directory also.

dervishe · Apr 21, 2023

I misread the original question

The permissions of the 2 directories are: 755 owned by root:wheel

Alain De Vos · Apr 21, 2023

Code:

chown mysql:mysql /var/db/mysql
chmod 751 /var/db/mysql
chmod 755 /var/db
chmod 755 /var

ltning · Apr 21, 2023

And /?

dervishe · Apr 21, 2023

Actually the permissions / owners are:
/ 700 root:wheel
/var 755 root:wheel
/var/db 755 root:wheel
/var/db/mysql 755 mysql:mysql
/var/db/mysql/data 755 mysql:mysql
/var/db/mysql/log 755 mysql:mysql

Alain De Vos · Apr 21, 2023

Code:

chown root:wheel /
chmod 755 /

dervishe · Apr 21, 2023

Alain De Vos said:
Code:

chown root:wheel / chmod 755 /

ltning said:
And /?

You got it !
Thx a lot to both of you !

It was that... 700 instead of 755
I don't know why i didn't saw it before...

Solved Jails, permissions issues

dervishe

SirDice

Administrator

dervishe

SirDice

Administrator

dervishe

SirDice

Administrator

dervishe

SirDice

Administrator

dervishe

SirDice

Administrator

Emrion

dervishe

SirDice

Administrator

dervishe

freebuser

dervishe

ltning

Alain De Vos

dervishe

Alain De Vos

ltning

dervishe

Alain De Vos

dervishe