/usr/bin/ssh Segfault

adams

adams

I swear I ran into this before and I cannot for the life of my remember what the solution was.

I have OpenSSH portable installed on all our servers, and /usr/local/bin/ssh works fine. But when running /usr/bin/ssh this happens on just a few servers starting a few days ago (I suspect a pkg(8) update or possibly one of the freebsd-updates):

Code: 
Segmentation fault: 11 (core dumped)

All the OSes are completely updated to -STABLE, kernel: FreeBSD 9.1-RELEASE-p13 #4: Thu May 15 16:48:10 EDT 2014

The really odd thing is I have lots of other servers that don't have this issue, just a few that do. All are updated. When running truss side by side I can see this is where things diverge.

On a server that segfaults:

Code: 
munmap(0x800646000,4096)                         = 0 (0x0)
close(4)                                         = 0 (0x0)
access("/usr/local/lib/libssl.so.8",0)           = 0 (0x0)
open("/usr/local/lib/libssl.so.8",O_RDONLY,040765300) = 4 (0x4)

On a server that has no issue:

Code: 
munmap(0x800646000,4096)                         = 0 (0x0)
close(4)                                         = 0 (0x0)
access("/usr/local/lib/libssl.so.6",0)           ERR#2 'No such file or directory'
access("/usr/lib/libssl.so.6",0)                 = 0 (0x0)
open("/usr/lib/libssl.so.6",O_RDONLY,040765300)  = 4 (0x4)

So for some reason /usr/bin/ssh is loading the /usr/local/lib version of libssl instead of /usr/lib which I'm guessing is the root of the issue. Anyone seen this before or have any ideas? I'm kicking myself as I know I ran into this before but can't remember what the solution was and didn't document it.
 
Do a ldd /usr/bin/ssh to confirm that it links to non-base libSSL. Also do ldconfig -r and paste output here.
 
(I renamed it ssh.orig as I'm symlinking it to the /usr/local/bin/ssh copy to keep people working in the mean time)

Code: 
>>>>  ldd /usr/bin/ssh.orig
/usr/bin/ssh.orig:
        libssh.so.5 => /usr/lib/libssh.so.5 (0x80083f000)
        libutil.so.9 => /lib/libutil.so.9 (0x800a90000)
        libz.so.6 => /lib/libz.so.6 (0x800ca3000)
        libgssapi.so.10 => /usr/lib/libgssapi.so.10 (0x800eb7000)
        libcrypt.so.5 => /lib/libcrypt.so.5 (0x8010c0000)
        libcrypto.so.6 => /lib/libcrypto.so.6 (0x8012df000)
        libc.so.7 => /lib/libc.so.7 (0x801680000)
        libkrb5.so.10 => /usr/lib/libkrb5.so.10 (0x8019d3000)
        libhx509.so.10 => /usr/lib/libhx509.so.10 (0x801c43000)
        libasn1.so.10 => /usr/lib/libasn1.so.10 (0x801e83000)
        libcom_err.so.5 => /usr/lib/libcom_err.so.5 (0x802105000)
        libmd.so.5 => /lib/libmd.so.5 (0x802307000)
        libroken.so.10 => /usr/lib/libroken.so.10 (0x802517000)

So it doesn't actually link to non-base ... which makes me wonder even more.

Code: 
>>>>  ldconfig -r
/var/run/ld-elf.so.hints:
        search directories: /lib:/usr/lib:/usr/lib/compat:/usr/local/lib:/usr/local/lib/pth
        0:-lc.7 => /lib/libc.so.7
        1:-lcrypt.5 => /lib/libcrypt.so.5
        2:-lkvm.5 => /lib/libkvm.so.5
        3:-lm.5 => /lib/libm.so.5
        4:-lmd.5 => /lib/libmd.so.5
        5:-lncurses.8 => /lib/libncurses.so.8
        6:-lncursesw.8 => /lib/libncursesw.so.8
        7:-lsbuf.6 => /lib/libsbuf.so.6
        8:-lutil.9 => /lib/libutil.so.9
        9:-lalias.7 => /lib/libalias.so.7
        10:-lbegemot.4 => /lib/libbegemot.so.4
        11:-lcam.6 => /lib/libcam.so.6
        12:-ldevstat.7 => /lib/libdevstat.so.7
        13:-ledit.7 => /lib/libedit.so.7
        14:-lbsdxml.4 => /lib/libbsdxml.so.4
        15:-lgeom.5 => /lib/libgeom.so.5
        16:-lipsec.4 => /lib/libipsec.so.4
        17:-lipx.5 => /lib/libipx.so.5
        18:-ljail.1 => /lib/libjail.so.1
        19:-lkiconv.4 => /lib/libkiconv.so.4
        20:-lpcap.8 => /lib/libpcap.so.8
        21:-lthr.3 => /lib/libthr.so.3
        22:-lufs.6 => /lib/libufs.so.6
        23:-lulog.0 => /lib/libulog.so.0
        24:-lz.6 => /lib/libz.so.6
        25:-lavl.2 => /lib/libavl.so.2
        26:-lctf.2 => /lib/libctf.so.2
        27:-ldtrace.2 => /lib/libdtrace.so.2
        28:-lnvpair.2 => /lib/libnvpair.so.2
        29:-lumem.2 => /lib/libumem.so.2
        30:-luutil.2 => /lib/libuutil.so.2
        31:-lzfs.2 => /lib/libzfs.so.2
        32:-lzpool.2 => /lib/libzpool.so.2
        33:-lgcc_s.1 => /lib/libgcc_s.so.1
        34:-lreadline.8 => /lib/libreadline.so.8
        35:-lssp.0 => /lib/libssp.so.0
        36:-lcrypto.6 => /lib/libcrypto.so.6
        37:-lbsm.3 => /usr/lib/libbsm.so.3
        38:-lauditd.5 => /usr/lib/libauditd.so.5
        39:-lcom_err.5 => /usr/lib/libcom_err.so.5
        40:-lelf.1 => /usr/lib/libelf.so.1
        41:-lform.5 => /usr/lib/libform.so.5
        42:-lmenu.5 => /usr/lib/libmenu.so.5
        43:-lpanel.5 => /usr/lib/libpanel.so.5
        44:-lformw.5 => /usr/lib/libformw.so.5
        45:-lmenuw.5 => /usr/lib/libmenuw.so.5
        46:-lpanelw.5 => /usr/lib/libpanelw.so.5
        47:-lnetgraph.4 => /usr/lib/libnetgraph.so.4
        48:-lradius.4 => /usr/lib/libradius.so.4
        49:-lrpcsvc.5 => /usr/lib/librpcsvc.so.5
        50:-ltacplus.5 => /usr/lib/libtacplus.so.5
        51:-lypclnt.4 => /usr/lib/libypclnt.so.4
        52:-larchive.5 => /usr/lib/libarchive.so.5
        53:-lBlocksRuntime.0 => /usr/lib/libBlocksRuntime.so.0
        54:-lbluetooth.4 => /usr/lib/libbluetooth.so.4
        55:-lbsnmp.6 => /usr/lib/libbsnmp.so.6
        56:-lbz2.4 => /usr/lib/libbz2.so.4
        57:-lcalendar.5 => /usr/lib/libcalendar.so.5
        58:-ldevinfo.5 => /usr/lib/libdevinfo.so.5
        59:-ldwarf.3 => /usr/lib/libdwarf.so.3
        60:-lfetch.6 => /usr/lib/libfetch.so.6
        61:-lftpio.8 => /usr/lib/libftpio.so.8
        62:-lgpib.3 => /usr/lib/libgpib.so.3
        63:-lgssapi.10 => /usr/lib/libgssapi.so.10
        64:-lrpcsec_gss.1 => /usr/lib/librpcsec_gss.so.1
        65:-llzma.5 => /usr/lib/liblzma.so.5
        66:-lmagic.4 => /usr/lib/libmagic.so.4
        67:-lmemstat.3 => /usr/lib/libmemstat.so.3
        68:-lmilter.5 => /usr/lib/libmilter.so.5
        69:-lmp.7 => /usr/lib/libmp.so.7
        70:-lncp.4 => /usr/lib/libncp.so.4
        71:-lngatm.4 => /usr/lib/libngatm.so.4
        72:-lopie.7 => /usr/lib/libopie.so.7
        73:-lpam.5 => /usr/lib/libpam.so.5
        74:-lpmc.5 => /usr/lib/libpmc.so.5
        75:-lproc.2 => /usr/lib/libproc.so.2
        76:-lprocstat.1 => /usr/lib/libprocstat.so.1
        77:-lrt.1 => /usr/lib/librt.so.1
        78:-lrtld_db.2 => /usr/lib/librtld_db.so.2
        79:-lsdp.4 => /usr/lib/libsdp.so.4
        80:-lsmb.4 => /usr/lib/libsmb.so.4
        81:-lstdbuf.1 => /usr/lib/libstdbuf.so.1
        82:-lthread_db.3 => /usr/lib/libthread_db.so.3
        83:-lugidfw.4 => /usr/lib/libugidfw.so.4
        84:-lusbhid.4 => /usr/lib/libusbhid.so.4
        85:-lusb.2 => /usr/lib/libusb.so.2
        86:-lvgl.6 => /usr/lib/libvgl.so.6
        87:-lwrap.6 => /usr/lib/libwrap.so.6
        88:-llwres.80 => /usr/lib/liblwres.so.80
        89:-ldialog.7 => /usr/lib/libdialog.so.7
        90:-lgomp.1 => /usr/lib/libgomp.so.1
        91:-lodialog.7 => /usr/lib/libodialog.so.7
        92:-lgnuregex.5 => /usr/lib/libgnuregex.so.5
        93:-lhistory.8 => /usr/lib/libhistory.so.8
        94:-lstdc++.6 => /usr/lib/libstdc++.so.6
        95:-lsupc++.1 => /usr/lib/libsupc++.so.1
        96:-lasn1.10 => /usr/lib/libasn1.so.10
        97:-lgssapi_krb5.10 => /usr/lib/libgssapi_krb5.so.10
        98:-lgssapi_ntlm.10 => /usr/lib/libgssapi_ntlm.so.10
        99:-lgssapi_spnego.10 => /usr/lib/libgssapi_spnego.so.10
        100:-lhdb.10 => /usr/lib/libhdb.so.10
        101:-lheimntlm.10 => /usr/lib/libheimntlm.so.10
        102:-lhx509.10 => /usr/lib/libhx509.so.10
        103:-lkadm5clnt.10 => /usr/lib/libkadm5clnt.so.10
        104:-lkadm5srv.10 => /usr/lib/libkadm5srv.so.10
        105:-lkafs5.10 => /usr/lib/libkafs5.so.10
        106:-lkrb5.10 => /usr/lib/libkrb5.so.10
        107:-lroken.10 => /usr/lib/libroken.so.10
        108:-lssl.6 => /usr/lib/libssl.so.6
        109:-lssh.5 => /usr/lib/libssh.so.5
        110:-lbsnmptools.0 => /usr/lib/libbsnmptools.so.0
        111:-lcharset.1 => /usr/local/lib/libcharset.so.1
        112:-liconv.3 => /usr/local/lib/libiconv.so.3
        113:-lintl.9 => /usr/local/lib/libintl.so.9
        114:-lasprintf.0 => /usr/local/lib/libasprintf.so.0
        115:-lgettextpo.5 => /usr/local/lib/libgettextpo.so.5
        116:-llber-2.4.8 => /usr/local/lib/liblber-2.4.so.8
        117:-lldap-2.4.8 => /usr/local/lib/libldap-2.4.so.8
        118:-lldap_r-2.4.8 => /usr/local/lib/libldap_r-2.4.so.8
        119:-lpcre.3 => /usr/local/lib/libpcre.so.3
        120:-lpcre16.2 => /usr/local/lib/libpcre16.so.2
        121:-lpcre32.0 => /usr/local/lib/libpcre32.so.0
        122:-lpcreposix.0 => /usr/local/lib/libpcreposix.so.0
        123:-lpcrecpp.0 => /usr/local/lib/libpcrecpp.so.0
        124:-lcord.1 => /usr/local/lib/libcord.so.1
        125:-lgc.1 => /usr/local/lib/libgc.so.1
        126:-lgccpp.1 => /usr/local/lib/libgccpp.so.1
        127:-lidn.17 => /usr/local/lib/libidn.so.17
        128:-lldns.1 => /usr/local/lib/libldns.so.1
        129:-lcurl.7 => /usr/local/lib/libcurl.so.7
        130:-lpython3.3m.1 => /usr/local/lib/libpython3.3m.so.1
        131:-lxml2.2 => /usr/local/lib/libxml2.so.2
        132:-lgpg-error.0 => /usr/local/lib/libgpg-error.so.0
        133:-lgcrypt.19 => /usr/local/lib/libgcrypt.so.19
        134:-lxslt.2 => /usr/local/lib/libxslt.so.2
        135:-lexslt.8 => /usr/local/lib/libexslt.so.8
        136:-lgnutlsxx.27 => /usr/local/lib/libgnutlsxx.so.27
        137:-lpaper.2 => /usr/local/lib/libpaper.so.2
        138:-lpython2.7.1 => /usr/local/lib/libpython2.7.so.1
        139:-lexpat.6 => /usr/local/lib/libexpat.so.6
        140:-lestr.0 => /usr/local/lib/libestr.so.0
        141:-lee.0 => /usr/local/lib/libee.so.0
        142:-ljson.1 => /usr/local/lib/libjson.so.1
        143:-ljson-c.2 => /usr/local/lib/libjson-c.so.2
        144:-lgmp.10 => /usr/local/lib/libgmp.so.10
        145:-lgmpxx.4 => /usr/local/lib/libgmpxx.so.4
        146:-lhogweed.2 => /usr/local/lib/libhogweed.so.2
        147:-lnettle.4 => /usr/local/lib/libnettle.so.4
        148:-ltasn1.7 => /usr/local/lib/libtasn1.so.7
        149:-lffi.6 => /usr/local/lib/libffi.so.6
        150:-lp11-kit.0 => /usr/local/lib/libp11-kit.so.0
        151:-lgnutls-extra.26 => /usr/local/lib/libgnutls-extra.so.26
        152:-lgnutls-openssl.27 => /usr/local/lib/libgnutls-openssl.so.27
        153:-lgnutls.26 => /usr/local/lib/libgnutls.so.26
        154:-lrelp.0 => /usr/local/lib/librelp.so.0
        155:-ldnet.1 => /usr/local/lib/libdnet.so.1
        156:-lhgfs.0 => /usr/local/lib/libhgfs.so.0
        157:-liculx.52 => /usr/local/lib/libiculx.so.52
        158:-licutest.52 => /usr/local/lib/libicutest.so.52
        159:-licutu.52 => /usr/local/lib/libicutu.so.52
        160:-licuuc.52 => /usr/local/lib/libicuuc.so.52
        161:-licudata.52 => /usr/local/lib/libicudata.so.52
        162:-licui18n.52 => /usr/local/lib/libicui18n.so.52
        163:-licuio.52 => /usr/local/lib/libicuio.so.52
        164:-lglib-2.0.0 => /usr/local/lib/libglib-2.0.so.0
        165:-lgmodule-2.0.0 => /usr/local/lib/libgmodule-2.0.so.0
        166:-lgthread-2.0.0 => /usr/local/lib/libgthread-2.0.so.0
        167:-lgobject-2.0.0 => /usr/local/lib/libgobject-2.0.so.0
        168:-lgio-2.0.0 => /usr/local/lib/libgio-2.0.so.0
        169:-lvmtools.0 => /usr/local/lib/libvmtools.so.0
        170:-lguestlib.0 => /usr/local/lib/libguestlib.so.0
        171:-lassuan.0 => /usr/local/lib/libassuan.so.0
        172:-lksba.19 => /usr/local/lib/libksba.so.19
        173:-lpkg.1 => /usr/local/lib/libpkg.so.1
        174:-licule.52 => /usr/local/lib/libicule.so.52
        175:-llogging-stdlog.0 => /usr/local/lib/liblogging-stdlog.so.0
        176:-lcrypto.8 => /usr/local/lib/libcrypto.so.8
        177:-lssl.8 => /usr/local/lib/libssl.so.8
        178:-lsasl2.3 => /usr/local/lib/libsasl2.so.3
        179:-lspf2.3 => /usr/local/lib/libspf2.so.3
        180:-lpth.20 => /usr/local/lib/pth/libpth.so.20
        181:-lpthread.20 => /usr/local/lib/pth/libpthread.so.20
 
I can confirm that uninstalling the OpenSSL port fixes this, not clear if it's a symptom of an LD path order thing or actually something the port does (using openssl-1.0.1_12).
 
These servers use LDAP_NSS ... the issue only is a problem for LDAP-based users, so I suspect there is a conflict between openldap-client's SSL (we have WITH_OPENSSL_PORT=YES set) and the base somehow.
 
You must log in or register to reply here.
Back
Top