OpenLDAP causing cron to core dump

T

Ted

I recently installed an OpenLDAP server and was setting up the client nodes when we realized that it was causing cron to core dump and stop running.

Code: 
Jan 29 08:27:00 <server-name> kernel: pid 61506 (cron), uid 0: exited on signal 11 (core dumped)

I followed this walkthrough https://www.freebsd.org/doc/en/articles/ldap-auth/index.html to setup the server and client nodes, the server and clients are running the latest FreeBSD 10.2.

Has anyone ever heard of this before or what might be causing it? Or even point me in a direction as to where to look for the problem, any guidance is appreciated.
 
Knows nothing about LDAP, just some generic troubleshooting:

Are there multiple pam libraries ? find / -name "libpam.so*"
Does
Code: 
# gdb /us/sbin/cron cron.core
where
reveal anything ?
Anything alarming in
Code: 
# killall cron
# cron -x bit,ext,load,misc,pars,proc,sch
Code: 
# ktrace -d cron # descendants
...wait...
# kdump

Juha
 
Juha,

When we ran the find command we found two libpam files:

Code: 
/usr/obj/usr/src/tmp/usr/lib/libpam.so.5
/usr/obj/usr/src/tmp/usr/lib/libpam.so
/usr/obj/usr/src/lib/libpam/libpam/libpam.so
/usr/obj/usr/src/lib/libpam/libpam/libpam.so.5
/usr/obj/usr/src/lib32/usr/lib32/libpam.so.5
/usr/obj/usr/src/lib32/usr/lib32/libpam.so
/usr/obj/lib32/usr/src/lib/libpam/libpam/libpam.so
/usr/obj/lib32/usr/src/lib/libpam/libpam/libpam.so.5
/usr/lib/libpam.so
/usr/lib/libpam.so.5
/usr/lib32/libpam.so.5
/usr/lib32/libpam.so

When we ran
Code: 
cron -x bit,ext,load,misc,pars,proc,sch
some alarming output was:

Code: 
load_entry()...about to parse username
load_entry()...got root
Segmentation fault (core dumped)

The kdump file is attached as a text file.

Is there anything that stands out to you that might cause cron to core dump?

Thanks for any help that you might be able to provide.
 

Attachments

Ugh, sorry. I goofed. It's ktrace -i cron, not -d.

Juha

Code: 
$ ldd /usr/sbin/cron
/usr/sbin/cron:
  libpam.so.5 => /usr/lib/libpam.so.5 (0x28070000)
  libutil.so.9 => /lib/libutil.so.9 (0x2807b000)
  libc.so.7 => /lib/libc.so.7 (0x2808d000)
$ ldd /usr/lib/pam_unix.so
/usr/lib/pam_unix.so:
  libpam.so.5 => /usr/lib/libpam.so.5 (0x281f4000)
  libutil.so.9 => /lib/libutil.so.9 (0x281ff000)
  libcrypt.so.5 => /lib/libcrypt.so.5 (0x28211000)
  libypclnt.so.4 => /usr/lib/libypclnt.so.4 (0x28235000)
  libc.so.7 => /lib/libc.so.7 (0x2806c000)
 
Thanks for the responses, we are looking over the kdump right now, but I wanted to let you know when I ran the ldd commands, we got the same results.

Will let you know what we find.

Thanks.
 
So here is the kdump that we have from ktrace -i cron, do you see anything that stands out?

Thanks again.
 

Attachments

So it fails in getpwnam(), it seems. It can't find nss_files.so for "passwd: files ldap" name service, but the builtin might be enough.

*scratch* Doesn't anything else coredump?
Juha

Code: 
...
"/usr/lib/libpam.so.5"
"/lib/libutil.so.9"
"/lib/libc.so.7"

fork setsid
write pidfile
...
read /etc/crontab
read login.conf.db did login_getclass() ?
read nsswitch.conf  does getpwnam() ?

"/lib/nss_files.so.1"
"/usr/lib/nss_files.so.1"
"/usr/lib/compat/nss_files.so.1"
"/usr/local/lib/nss_files.so.1"
"/lib/nss_files.so.1"
"/usr/lib/nss_files.so.1" all missing

"/usr/local/lib/nss_ldap.so.1" these were found and mapped
"/usr/local/lib/libldap-2.4.so.2"
"/usr/local/lib/liblber-2.4.so.2"
"/usr/lib/libkrb5.so.11"
"/usr/lib/libcom_err.so.5"
"/usr/lib/libgssapi_krb5.so.10"
"/usr/local/lib/libssl.so.8"
"/usr/local/lib/libcrypto.so.8"
"/usr/lib/libasn1.so.11"
"/lib/libcrypt.so.5"
"/lib/libcrypto.so.7"
"/usr/lib/libhx509.so.11"
"/usr/lib/libroken.so.11"
"/usr/lib/libwind.so.11"
"/usr/lib/libheimbase.so.11"
"/usr/lib/private/libheimipcc.so.11"
"/usr/lib/libgssapi.so.10"
"/lib/libthr.so.3"

coredump
 
So far the only service to coredump was cron(8).

The /etc/nsswitch.conf file was:

Code: 
group: compat
passwd: compat

In order to have LDAP work we changed it to:

Code: 
group: files ldap
passwd: files ldap

As soon as I remove ldap, cron(8) works fine, but then in the process breaks LDAP.
 
Between the library loads and the SIGSEGV there is a short interlude, it queries kern.usrstack and RLIMIT_STACK, then mprotects (successfully) 0.5GB of stack read, write, execute. Maybe this is as it should be, but being just before the crash it feels bad.

hands raised,
Juha
 
  • Thanks
Reactions: Ted
Juha,

So I am attaching two dump files, one from our server that is running LDAP and cron is crashing (ldap) and the other (noldap) that is not running ldap and cron is not crashing. I have done a diff on these two files but nothing is glaring at me as to point to the issue other than nsswitch.conf file, turns LDAP on and off so to speak.

Any ideas?

Thanks.
 

Attachments

There not much in the noldap trace, it does not reach the problem point. "Already running".

Debugging (with or without the cron.core) is not for humans, there are no symbols. If you have base system sources, try
Code: 
# cd /usr/src/usr.sbin/cron/
# make
# gdb ./cron/cron
(gdb) run -x bit,ext,load,misc,pars,proc,sch
...
(gdb) where

Juha
 
Hi!

Your trouble in conflict of libcrypto libraries:

"/usr/local/lib/libcrypto.so.8"
"/lib/libcrypto.so.7"

Your cron used libcrypto.7, and your nss_ldap and pam_ldap (and openldap-client) use libcrypto.8 - it's trouble.

You need to rebuild openldap, nss and pam with openssl version from base system:

SSL_DEFAULT=base

Do not forget about sources of your system (/usr/src) before build :) It's need for build with "base" openssl!
 
Hi Oleg,

What it turned out to be was I needed to install the package nss_pam_ldapd and enable it on start up and that corrected the issue.

Thanks for your input though, when I have a spare client I will try your method.
 
  • Thanks
Reactions: Oko
You must log in or register to reply here.
Back
Top