mosquitto used to run fine, now exiting - how to debug?

[garrett]

Hi,

I am running mosquitto in a jail, and previously it was working fine. Now it shows an error in the host messages log on jail startup:

kernel: pid 64031 (mosquitto), jid 9, uid 10001: exited on signal 11 (no core dump - other error)

From the jail, the config is still testing as ok:

# mosquitto -c ./mosquitto.conf --test-config
1771794490: The 'per_listener_settings' option is now deprecated and will be removed in version 3.0. Please see the documentation for how to achieve the same effect.
1771794490: Configuration file is OK.
1771794490: mosquitto version 2.1.2 terminating

When I run service mosquitto start it appears to be running:

# service mosquitto start
Starting mosquitto.
1771794773: The 'per_listener_settings' option is now deprecated and will be removed in version 3.0. Please see the documentation for how to achieve the same effect.
1771794773: Info: running mosquitto as user: mosquitto.

There is a process file generated, but apparently it does not get removed after the service stops:

 # ls -al /var/run/mosquitto
total 10
drwxrwxr-x  2 mosquitto mosquitto  3 Feb 22 21:12 .
drwxr-xr-x  9 root      wheel     26 Feb 22 21:12 ..
-rw-r--r--  1 mosquitto mosquitto  5 Feb 22 21:12 mosquitto.pid

# service mosquitto status
mosquitto is not running.

If I try running as root:

# mosquitto -c /usr/local/etc/mosquitto/mosquitto.conf
1771795342: The 'per_listener_settings' option is now deprecated and will be removed in version 3.0. Please see the documentation for how to achieve the same effect.
1771795342: Info: running mosquitto as user: mosquitto.
Segmentation fault         mosquitto -c /usr/local/etc/mosquitto/mosquitto.co

... the corresponding host log is:

kernel: pid 64845 (mosquitto), jid 9, uid 10001: exited on signal 11 (no core dump - sugid process denied by kern.sugid_coredump)

If I try manually running as mosquitto user:

# su -m mosquitto -c "mosquitto -c /usr/local/etc/mosquitto/mosquitto.conf"
1771795463: The 'per_listener_settings' option is now deprecated and will be removed in version 3.0. Please see the documentation for how to achieve the same effect.
1771795463: Info: running mosquitto as user: mosquitto.

... but same error on host:

kernel: pid 64855 (mosquitto), jid 9, uid 10001: exited on signal 11 (no core dump - other error)

Plus, there is no output in /var/log/mosquitto.log and log type is set to debug.

I am assuming an update broke something at some point - nothing else was changed. Maybe I missed the memo on a change needed. I did try reinstalling but same problems. Last time I know for sure it worked was sometime around 6 months ago.

Does anybody know what might be causing this?

 

[garrett]

If I run as root but do not reference the config file location, it starts, and says it's running as user "mosquitto":

 # mosquitto
1771811568: Info: running mosquitto as user: mosquitto.
1771811568: mosquitto version 2.1.2 starting
1771811568: Using default config.
1771811568: Bridge support available.
1771811568: Persistence support available.
1771811568: TLS support available.
1771811568: TLS-PSK support available.
1771811568: Websockets support available.
1771811568: Starting in local only mode. Connections will only be possible from clients running on this machine.
1771811568: Create a configuration file which defines a listener to allow remote access.
1771811568: For more details see https://mosquitto.org/documentation/authentication-methods/
1771811568: Opening ipv4 listen socket on port 1883.
1771811568: Opening ipv6 listen socket on port 1883.
1771811568: Warning: Protocol not supported
1771811568: mosquitto version 2.1.2 running
^C1771811578: mosquitto version 2.1.2 terminating

Protocol is not supported... probably ipv6. But, it starts. Must be something going on with the config file.

EDIT: I've narrowed it down to the acl_file and password_file. It runs without them, but if they are noted in the config and NOT owned by the mosquitto user, it will throw an error that they can't be read. If they ARE owned by the mosquitto user, it results in a segfault.