jails MariaDB in Bastille jail with jailed datasets. Jailed datasets are not mounted on jail startup on boot time.

W

willyvmm

I have a Bastille Jail with some jailed zfs datasets for mariadb.
Jails are created using Bastille, datasets are jailed using Bastille zfs command.

The problem is when the Jail starts, and try to start mysql-server service, seems like the ZFS datasets are not ready/mounted inside the jail.

This is log (/var/log/mysql/mysqld.err) when it try try to start on boot up:
Code: 
2025-11-05 14:42:47 0 [ERROR] InnoDB: File /var/db/mysql/innodb/data/ibdata1 was not found
2025-11-05 14:42:47 0 [ERROR] InnoDB: Operating system error number 2 in a file operation.
2025-11-05 14:42:47 0 [ERROR] InnoDB: Error number 2 means 'No such file or directory'
2025-11-05 14:42:47 0 [ERROR] InnoDB: Cannot open datafile '/var/db/mysql/innodb/data/ibdata1'
2025-11-05 14:42:47 0 [ERROR] InnoDB: Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data!
2025-11-05 14:42:47 0 [ERROR] InnoDB: Database creation was aborted with error Cannot open a file. You may need to delete the ibdata1 file before trying to start up again.
2025-11-05 14:42:48 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2025-11-05 14:42:48 0 [Warning] 'wsrep-causal-reads' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
2025-11-05 14:42:48 0 [ERROR] Unknown/unsupported storage engine: innodb
2025-11-05 14:42:48 0 [ERROR] Aborting
Looks like the /var/db/mysql/innodb/data dataset is not yet mounted.

When I start the service later manually: (service mysql-server start)
it starts flawless:
Code: 
2025-11-05 15:09:56 0 [Note] Starting MariaDB 11.8.3-MariaDB-log source revision b565b3e7e041c480fd8013e3beec6b99544d6bf8 server_uid Go3FyZZup3K9IH7uKk3GKxTcEXw= as process 5543
2025-11-05 15:09:56 0 [Note] InnoDB: Compressed tables use zlib 1.3.1
2025-11-05 15:09:56 0 [Note] InnoDB: Number of transaction pools: 1
2025-11-05 15:09:56 0 [Note] InnoDB: Using AVX512 instructions
2025-11-05 15:09:56 0 [Note] InnoDB: innodb_buffer_pool_size_max=128m, innodb_buffer_pool_size=128m
2025-11-05 15:09:56 0 [Note] InnoDB: Completed initialization of buffer pool
2025-11-05 15:09:56 0 [Note] InnoDB: End of log at LSN=49982
2025-11-05 15:09:56 0 [Note] InnoDB: Opened 3 undo tablespaces
2025-11-05 15:09:56 0 [Note] InnoDB: 128 rollback segments in 3 undo tablespaces are active.
2025-11-05 15:09:56 0 [Note] InnoDB: Setting file '/var/db/mysql/innodb/data/ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
2025-11-05 15:09:56 0 [Note] InnoDB: File '/var/db/mysql/innodb/data/ibtmp1' size is now 12.000MiB.
2025-11-05 15:09:56 0 [Note] InnoDB: log sequence number 49982; transaction id 22
2025-11-05 15:09:56 0 [Note] Plugin 'FEEDBACK' is disabled.
2025-11-05 15:09:56 0 [Note] Plugin 'wsrep-provider' is disabled.
2025-11-05 15:09:56 0 [Note] InnoDB: Loading buffer pool(s) from /var/db/mysql/innodb/data/ib_buffer_pool
2025-11-05 15:09:56 0 [Warning] 'wsrep-causal-reads' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
2025-11-05 15:09:56 0 [Warning] 'wsrep-causal-reads' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
2025-11-05 15:09:56 0 [Note] InnoDB: Buffer pool(s) load completed at 251105 15:09:56
2025-11-05 15:09:57 0 [Note] Server socket created on IP: '0.0.0.0', port: '3306'.
2025-11-05 15:09:57 0 [Note] mariadbd: Event Scheduler: Loaded 0 events
2025-11-05 15:09:57 0 [Note] /usr/local/libexec/mariadbd: ready for connections.
Version: '11.8.3-MariaDB-log'  socket: '/var/run/mysql/mysql.sock'  port: 3306  FreeBSD Ports

Any idea why this happened?

There are also some strange infos when i manually start/stop the jail:

Code: 
willy@browar: ~ % sudo bastille stop mariadb

[mariadb]:
cannot unmount '/var/db/mysql/innodb/log': pool or dataset is busy
cannot unmount '/var/db/mysql/data': pool or dataset is busy
cannot unmount '/var/db/mysql/innodb/log': pool or dataset is busy
cannot unmount '/var/db/mysql/innodb/data': pool or dataset is busy
cannot unmount '/var/db/mysql/innodb/log': pool or dataset is busy
mariadb: removed

Code: 
willy@browar: ~ % sudo bastille start mariadb

[mariadb]:
e0a_mariadb
e0b_mariadb
mariadb: created
cannot unmount '/var/db/mysql': permission denied
cannot unmount '/var/db/mysql/data': permission denied
cannot unmount '/var/db/mysql/innodb': permission denied
cannot unmount '/var/db/mysql/innodb/data': permission denied
cannot unmount '/var/db/mysql/innodb/log': permission denied

After manual stop/start cycle mariaDB usually starts.

I think adding a delay would help, but I would like to understand why?
 
Ok.
Jailed datasets are mounted inside the Jail after the jail has started, and unmounted before it is stopped ... at least in BASTILLE. That makes the jailed datasets near useless, or require some dirty hacks, what I don't.

sh: 
    # Start jail
    jail ${OPTION} -f "${bastille_jailsdir}/${_jail}/jail.conf" -c "${_jail}"

    # Add ZFS jailed datasets
    if [ -s "${bastille_jailsdir}/${_jail}/zfs.conf" ]; then
        while read _dataset _mount; do
            zfs set jailed=on "${_dataset}"
            zfs jail ${_jail} "${_dataset}"
            jexec -l -U root "${_jail}" zfs set mountpoint="${_mount}" "${_dataset}"
            jexec -l -U root "${_jail}" zfs mount "${_dataset}" 2>/dev/null
        done < "${bastille_jailsdir}/${_jail}/zfs.conf"
    fi
 
Probably best to open an issue on the BastilleBSD repo on GitHub.

This is fairly new functionality, so I’m not sure how many people are using it currently.

In my testing it is necessary to unmount the datasets before the jail stops, otherwise it throws errors when trying to mount them again on jail startup.
 
The issue seems to be that on startup, Bastille uses the 'mount' command (which will attempt an unmount and remount if it is mounted) which fails due to not having the proper permissions.

Can you try commenting out lines 192 and 192 inside start.sh and also line 130 of stop.sh but only after you have run the 'bastille zfs TARGET jail' command exactly one time for a new dataset? DO NOT stop the jail before commenting out the above lines.

PS, we can take this to Github for more troubleshooting.
 
You must log in or register to reply here.
Back
Top