Unifi controller works fine but mongo keeps crashing(?) in jail

mca2 · Apr 12, 2024

I've setup a thin jail with its own VNET in FreeBSD14 for my unifi controller using quarterly packages

Code:

$ pkg query '%dn-%dv' unifi8-8.0.24
openjdk17-17.0.9+9.1
mongodb44-4.4.26
snappyjava-1.1.10.5

It works like a charm but my logs are filled with Mongo crashes. Any clue what goes wrong?

Code:

Apr 12 09:30:00 FreeBSD14 kernel: pid 12381 (mongod), jid 5, uid 975: exited on signal 6 (no core dump - other error)
Apr 12 09:35:00 FreeBSD14 kernel: pid 12954 (mongod), jid 5, uid 975: exited on signal 6 (no core dump - other error)
Apr 12 09:40:00 FreeBSD14 kernel: pid 14402 (mongod), jid 5, uid 975: exited on signal 6 (no core dump - other error)
Apr 12 09:45:00 FreeBSD14 kernel: pid 14988 (mongod), jid 5, uid 975: exited on signal 6 (no core dump - other error)
Apr 12 09:50:00 FreeBSD14 kernel: pid 16045 (mongod), jid 5, uid 975: exited on signal 6 (no core dump - other error)
Apr 12 09:55:00 FreeBSD14 kernel: pid 16614 (mongod), jid 5, uid 975: exited on signal 6 (no core dump - other error)
Apr 12 10:00:00 FreeBSD14 kernel: pid 17797 (mongod), jid 5, uid 975: exited on signal 6 (no core dump - other error)
Apr 12 10:05:00 FreeBSD14 kernel: pid 19696 (mongod), jid 5, uid 975: exited on signal 6 (no core dump - other error)
Apr 12 10:10:00 FreeBSD14 kernel: pid 20273 (mongod), jid 5, uid 975: exited on signal 6 (no core dump - other error)
Apr 12 10:15:00 FreeBSD14 kernel: pid 20960 (mongod), jid 5, uid 975: exited on signal 6 (no core dump - other error)
Apr 12 10:20:00 FreeBSD14 kernel: pid 22403 (mongod), jid 5, uid 975: exited on signal 6 (no core dump - other error)
Apr 12 10:25:00 FreeBSD14 kernel: pid 22795 (mongod), jid 5, uid 975: exited on signal 6 (no core dump - other error)

Hiroo Ono · Apr 12, 2024

Signal 6 is caused by abort(). Is there any messages in /var/db/mongodb/mongod.log?

mca2 · Apr 12, 2024

that folder is completely empty

Hiroo Ono · Apr 12, 2024

Well, you could try invoking mongod on the command line.

Stop the program that is running mongod every 5 minutes.
Run sudo -u mongodb -g mongodb mongod -v --config /usr/local/etc/mongodb.conf --dbpath /var/db/mongodb

mca2 · Apr 15, 2024

upgraded to the latest and 'greatest'

Code:

[pkg query '%dn-%dv' unifi8-8.1.113
openjdk17-17.0.10+7.1_1
mongodb50-5.0.26_1
snappyjava-1.1.10.5

and now it's even worse and unifi wont start at all

Code:

Apr 16 00:19:17 FreeBSD14 kernel: pid 23466 (mongod), jid 9, uid 975: exited on signal 4 (no core dump - other error)
Apr 16 00:19:20 FreeBSD14 kernel: pid 23693 (mongod), jid 9, uid 975: exited on signal 4 (no core dump - other error)
Apr 16 00:19:23 FreeBSD14 kernel: pid 24375 (mongod), jid 9, uid 975: exited on signal 4 (no core dump - other error)
Apr 16 00:19:26 FreeBSD14 kernel: pid 24401 (mongod), jid 9, uid 975: exited on signal 4 (no core dump - other error)
Apr 16 00:19:29 FreeBSD14 kernel: pid 24495 (mongod), jid 9, uid 975: exited on signal 4 (no core dump - other error)
Apr 16 00:19:32 FreeBSD14 kernel: pid 24871 (mongod), jid 9, uid 975: exited on signal 4 (no core dump - other error)
Apr 16 00:19:35 FreeBSD14 kernel: pid 25019 (mongod), jid 9, uid 975: exited on signal 4 (no core dump - other error)
Apr 16 00:19:38 FreeBSD14 kernel: pid 25085 (mongod), jid 9, uid 975: exited on signal 4 (no core dump - other error)
Apr 16 00:19:41 FreeBSD14 kernel: pid 25245 (mongod), jid 9, uid 975: exited on signal 4 (no core dump - other error)
Apr 16 00:19:44 FreeBSD14 kernel: pid 25586 (mongod), jid 9, uid 975: exited on signal 4 (no core dump - other error)
Apr 16 00:19:47 FreeBSD14 kernel: pid 25796 (mongod), jid 9, uid 975: exited on signal 4 (no core dump - other error)
Apr 16 00:19:50 FreeBSD14 kernel: pid 25924 (mongod), jid 9, uid 975: exited on signal 4 (no core dump - other error)
Apr 16 00:19:53 FreeBSD14 kernel: pid 25997 (mongod), jid 9, uid 975: exited on signal 4 (no core dump - other error)
Apr 16 00:19:56 FreeBSD14 kernel: pid 26334 (mongod), jid 9, uid 975: exited on signal 4 (no core dump - other error)
Apr 16 00:19:59 FreeBSD14 kernel: pid 26427 (mongod), jid 9, uid 975: exited on signal 4 (no core dump - other error)

So I killed unifi and ran the suggested code

 $ sudo -u mongodb -g mongodb mongod -v --config /usr/local/etc/mongodb.conf --dbpath /var/db/mongodb

which immediately returned with:

 Illegal instruction

and var/log/messages shows

Code:

Apr 16 00:27:00 FreeBSD14 kernel: pid 39166 (mongod), jid 9, uid 922: exited on signal 4 (no core dump - too large)
Apr 16 00:27:00 FreeBSD14 kernel: pid 39057 (sudo), jid 9, uid 0: exited on signal 4 (no core dump - bad address)

So I'm a bit lost here now

Hiroo Ono · Apr 15, 2024

It seems to be the same problem to this thread Thread net-mgmt-unifi8-upgrade-and-mongodb50-illegal-instruction-unresolved-symbols-avx.93105. Does your CPU support AVX instruction?

mca2 · Apr 15, 2024

sadly not

Code:

$ sysctl -a | egrep -i 'hw.machine|hw.model|hw.ncpu'
hw.machine: amd64
hw.model: Intel(R) Pentium(R) CPU G4560 @ 3.50GHz
hw.ncpu: 4
hw.machine_arch: amd64

Hiroo Ono · Apr 16, 2024

https://ark.intel.com/content/www/u...entium-processor-g4560-3m-cache-3-50-ghz.html
It seems that G4560 does not support AVX, so you have to unset AVX option of mongodb50 and build it from ports.

mca2 · Apr 16, 2024

I’ve always been warned not to mix binary packages and manual build ports. Any pointers where to start and not create a mess?

richardtoohey2 · Apr 16, 2024

Maybe build & test on another machine (or virtual machine). Then see if you can build your own packages and copy across to the AVX-less machine?

That way at least you don't pollute your packages-only machine if anything goes horribly wrong.

Hiroo Ono · Apr 16, 2024

I recommend to use poudriere.
https://wiki.freebsd.org/VladimirKrstulja/Guides/Poudriere
Just to use your own built packages for the jail would not require so many ports to build.

SirDice · Apr 16, 2024

mca2 said:
Any pointers where to start and not create a mess?

If you use the quarterly packages (that's the default on -RELEASE versions) make sure to checkout the quarterly branch of the ports tree. Or else you might run into significant version differences between quarterly and latest.

mca2 · Apr 26, 2024

Ok, that took a bit longer than expected, but poudriere worked and mongodb50 is running again, yet with still the same error

Code:

Apr 26 15:15:30 FreeBSD14 kernel: pid 42293 (mongod), jid 1, uid 975: exited on signal 6 (no core dump - other error)

Hiroo Ono said:
Well, you could try invoking mongod on the command line.

Stop the program that is running mongod every 5 minutes.

Run sudo -u mongodb -g mongodb mongod -v --config /usr/local/etc/mongodb.conf --dbpath /var/db/mongodb

I ran that and the log is full of these entries

Code:

{"t":{"$date":"2024-04-26T15:23:03.624+02:00"},"s":"D1", "c":"STORAGE",  "id":8097401, "ctx":"TimestampMonitor","msg":"No drop-pending idents have expired","attr":{"timestamp":{"$timestamp":{"t":0,"i":0}}}}
{"t":{"$date":"2024-04-26T15:23:03.630+02:00"},"s":"D1", "c":"INDEX",    "id":22533,   "ctx":"TTLMonitor","msg":"running TTL job for index","attr":{"namespace":"config.system.sessions","key":{"lastUse":1},"name":"lsidTTLIndex"}}
{"t":{"$date":"2024-04-26T15:23:03.630+02:00"},"s":"I",  "c":"INDEX",    "id":5479200, "ctx":"TTLMonitor","msg":"Deleted expired documents using index","attr":{"namespace":"config.system.sessions","index":"lsidTTLIndex","numDeleted":0,"durationMillis":0}}
{"t":{"$date":"2024-04-26T15:23:03.654+02:00"},"s":"I",  "c":"STORAGE",  "id":22430,   "ctx":"Checkpointer","msg":"WiredTiger message","attr":{"message":"[1714137783:654510][15675:0x1ed4f3a23000], WT_SESSION.checkpoint: [WT_VERB_CHECKPOINT_PROGRESS] saving ch
eckpoint snapshot min: 10, snapshot max: 10 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 667"}}
{"t":{"$date":"2024-04-26T15:23:04.624+02:00"},"s":"D1", "c":"STORAGE",  "id":8097401, "ctx":"TimestampMonitor","msg":"No drop-pending idents have expired","attr":{"timestamp":{"$timestamp":{"t":0,"i":0}}}}
{"t":{"$date":"2024-04-26T15:23:05.624+02:00"},"s":"D1", "c":"STORAGE",  "id":8097401, "ctx":"TimestampMonitor","msg":"No drop-pending idents have expired","attr":{"timestamp":{"$timestamp":{"t":0,"i":0}}}}
{"t":{"$date":"2024-04-26T15:23:06.624+02:00"},"s":"D1", "c":"STORAGE",  "id":8097401, "ctx":"TimestampMonitor","msg":"No drop-pending idents have expired","attr":{"timestamp":{"$timestamp":{"t":0,"i":0}}}}
{"t":{"$date":"2024-04-26T15:23:07.624+02:00"},"s":"D1", "c":"STORAGE",  "id":8097401, "ctx":"TimestampMonitor","msg":"No drop-pending idents have expired","attr":{"timestamp":{"$timestamp":{"t":0,"i":0}}}}
{"t":{"$date":"2024-04-26T15:23:08.624+02:00"},"s":"D1", "c":"STORAGE",  "id":8097401, "ctx":"TimestampMonitor","msg":"No drop-pending idents have expired","attr":{"timestamp":{"$timestamp":{"t":0,"i":0}}}}

But the 'no core dump' messages, dont appear in /var/log/messages

Hiroo Ono · Apr 26, 2024

Does this thread help?
Thread net-mgmt-unifi8-upgrade-and-mongodb50-illegal-instruction-unresolved-symbols-avx.93105

mca2 · Apr 26, 2024

Hiroo Ono said:
Does this thread help?
Thread net-mgmt-unifi8-upgrade-and-mongodb50-illegal-instruction-unresolved-symbols-avx.93105

it did, that's why I rebuilt disabling AVX, otherwise it wouldn't even come to live

pming · Apr 26, 2024

mca2 said:
it did, that's why I rebuilt disabling AVX, otherwise it wouldn't even come to live

It crashes regularly every 5 minutes
Apr 12 09:30:00 FreeBSD14 kernel: pid 12381 (mongod), jid 5, uid 975: exited on signal 6 (no core dump - other error)
Apr 12 09:35:00 FreeBSD14 kernel: pid 12954 (mongod), jid 5, uid 975: exited on signal 6 (no core dump - other error)
Apr 12 09:40:00 FreeBSD14 kernel: pid 14402 (mongod), jid 5, uid 975: exited on signal 6 (no core dump - other error)
Apr 12 09:45:00 FreeBSD14 kernel: pid 14988 (mongod), jid 5, uid 975: exited on signal 6 (no core dump - other error)
Apr 12 09:50:00
Apr 12 09:55:00
Apr 12 10:00:00
Apr 12 10:05:00

Why?

Do you have WiFi devices configured with this already?

mca2 · Apr 26, 2024

pming said:
It crashes regularly every 5 minutes
Apr 12 09:30:00 FreeBSD14 kernel: pid 12381 (mongod), jid 5, uid 975: exited on signal 6 (no core dump - other error)
Apr 12 09:35:00 FreeBSD14 kernel: pid 12954 (mongod), jid 5, uid 975: exited on signal 6 (no core dump - other error)
Apr 12 09:40:00 FreeBSD14 kernel: pid 14402 (mongod), jid 5, uid 975: exited on signal 6 (no core dump - other error)
Apr 12 09:45:00 FreeBSD14 kernel: pid 14988 (mongod), jid 5, uid 975: exited on signal 6 (no core dump - other error)
Apr 12 09:50:00
Apr 12 09:55:00
Apr 12 10:00:00
Apr 12 10:05:00

Why?

Do you have WiFi devices configured with this already?

yeah, all works otherwise fine. I don't experience any issues with the unifi controller

mca2 · Apr 27, 2024

I've been told Unifi8 only supports MongoDB3.6, 4.4 or 7.0. I'll see if that solves the issue

Unifi controller works fine but mongo keeps crashing(?) in jail

Administrator