Solved Postfix does not start/build anymore since upgrade to OpenSSL 1.0.2

pvoigt

Member

Reaction score: 5
Messages: 82

I am currently using 10.1-RELEASE (amd64) and have just upgraded openssl to version 1.0.2. Now Postfix 2.11.4,1 does not start anymore:
Code:
# service postfix onestart
postconf: environment corrupt; missing value for readme_d�
/usr/local/sbin/postconf: fatal: out of memory
postconf: environment corrupt; missing value for queue_di�
/usr/local/sbin/postconf: fatal: out of memory
postconf: environment corrupt; missing value for readme_d�
/usr/local/sbin/postconf: fatal: out of memory
postconf: environment corrupt; missing value for readme_d�
/usr/local/sbin/postconf: fatal: out of memory
postconf: environment corrupt; missing value for readme_d�
/usr/local/sbin/postconf: fatal: out of memory
postfix/postfix-script: fatal: unable to create missing queue directories
postfix/postfix-script: fatal: Postfix integrity check failed!
The same error appears during build:
Code:
# portmaster --no-confirm --no-term-title -D -G postfix
...
[src/tlsproxy]
[src/posttls-finger]
/bin/sh postfix-install -non-interactive -package
postconf: environment corrupt; missing value for readme_d�
bin/postconf: fatal: out of memory
postconf: environment corrupt; missing value for readme_d�
bin/postconf: fatal: out of memory
postconf: environment corrupt; missing value for readme_d�
bin/postconf: fatal: out of memory
postconf: environment corrupt; missing value for readme_d�
bin/postconf: fatal: out of memory
postconf: environment corrupt; missing value for readme_d�
bin/postconf: fatal: out of memory
postfix-install: Error: "" should be an absolute path name.
*** Error code 1

Stop.
make[2]: stopped in /usr/ports/mail/postfix/work/postfix-2.11.4
*** Error code 1

Stop.
make[1]: stopped in /usr/ports/mail/postfix
*** Error code 1

Stop.
make: stopped in /usr/ports/mail/postfix

===>>> make stage failed for mail/postfix
===>>> Aborting update


===>>> You can restart from the point of failure with this command line:
  portmaster <flags> mail/postfix
Well, I could rebuild all other ports depending on OpenSSL. I strongly suppose postfix needs a fix. Can anybody reproduce this issue. Should I open a PR for it?

Regards,
Peter
 
OP
OP
P

pvoigt

Member

Reaction score: 5
Messages: 82

Hm, I have just deinstalled and tried to reinstall - same effect. Here are my options:
Code:
# cd /usr/ports/mail/postfix
# make showconfig
===> The following configuration options are available for postfix-2.11.4,1:
  BDB=on: Berkeley DB (uses WITH_BDB_VER)
  CDB=off: CDB maps lookups
  DOCS=on: Build and/or install documentation
  INST_BASE=off: Install into /usr and /etc/postfix
  LDAP_SASL=on: OpenLDAP client-to-server SASL auth
  LMDB=off: LMDB maps
  MYSQL=off: MySQL maps (uses WITH_MYSQL_VER)
  NIS=off: NIS maps lookups
  OPENLDAP=on: OpenLDAP maps (uses WITH_OPENLDAP_VER)
  PCRE=on: Perl Compatible Regular Expressions
  PGSQL=off: PostgreSQL maps (uses DEFAULT_PGSQL_VER)
  SASL2=on: Cyrus SASLv2 (Simple Auth. and Sec. Layer)
  SPF=off: SPF support (via libspf2 1.2.x)
  SQLITE=off: SQLite maps
  TEST=off: SMTP/LMTP test server and generator
  TLS=on: SSL and TLS support
  VDA=off: VDA (Virtual Delivery Agent 32Bit)
====> Dovecot SASL authentication methods: you can only select none or one of them
  DOVECOT=off: Dovecot 1.x SASL authentication method
  DOVECOT2=on: Dovecot 2.x SASL authentication method
====> Kerberos network authentication protocol type: you can only select none or one of them
  SASLKRB5=off: If your SASL req. Kerberos5, select this
  SASLKMIT=off: If your SASL req. MIT Kerberos5, select this
===> Use 'make config' to modify these settings
And:
Code:
# cd /usr/ports/security/openssl
# make showconfig
===> The following configuration options are available for openssl-1.0.2:
  SHARED=on: build of shared libs
  THREADS=on: Threading support
  I386=off: Optimize for i386 (instead of i486+)
  SSE2=on: runtime SSE2 detection
  ASM=on: optimized Assembler code
  PADLOCK=off: VIA Padlock support
  ZLIB=on: zlib compression support
  GMP=off: gmp support (LGPLv3)
  SCTP=on: SCTP protocol support
  SSL2=on: SSLv2 protocol support
  SSL3=on: SSLv3 protocol support
  RFC3779=off: RFC3779 support (BGP)
  MD2=on: MD2 hash (obsolete)
  RC5=off: RC5 cipher (patented)
  EXPCIPHERS=off: Include experimental ciphers
  DOCS=on: Build and/or install documentation
  EC=on: Optimize NIST elliptic curves
===> Use 'make config' to modify these settings
I see nothing unusual so far.

I would like this get solved as soon as possible, because I am without any MTA and see not even any temporarily workaround.

Regards,
Peter
 

obsigna

Aspiring Daemon

Reaction score: 558
Messages: 956

At the first place, your postconf, this is the utility that checks the consistency of the configuration files, complains about:
Code:
postconf: environment corrupt; missing value for readme_dir
...
Did you already check the corresponding configuration file /usr/local/etc/postfix/main.cf? Does it contain a reasonable entry for readme_dir?

Mine does:
Code:
# POSTFIX LOCATIONS, USER, GROUP, AND PROTOCOL
readme_directory  = /usr/local/share/doc/postfix
...
My best educated guess is, that somehow your postfix configuration files became spoiled, and this coincidentally happened when OpenSSL got upgraded.
 
OP
OP
P

pvoigt

Member

Reaction score: 5
Messages: 82

Well, thanks so far for all feedback, but I am afraid I am having more serious problems since one of the yesterday port updates. My zsh is complaining about corrupt environment as well and some of my services are core dumping:
Code:
Mar 22 14:50:10 spock kernel: Trying to mount root from ufs:/dev/gpt/root [rw]...
Mar 22 14:51:17 spock kernel: pid 917 (sh), uid 0: exited on signal 11 (core dumped)
Mar 22 14:51:31 spock kernel: pid 1293 (dbus-daemon-launch-), uid 0: exited on signal 11
Mar 22 14:51:31 spock kernel: pid 1294 (polkit-read-auth-he), uid 0: exited on signal 11
Mar 22 14:51:34 spock kernel: pid 1430 (dbus-daemon-launch-), uid 0: exited on signal 11
Mar 22 14:51:34 spock kernel: pid 1365 (cron), uid 0: exited on signal 11 (core dumped)
Mar 22 14:51:34 spock kernel: pid 1428 (hald), uid 0: exited on signal 11 (core dumped)
Mar 22 14:53:15 spock kernel: pid 1493 (cron), uid 0: exited on signal 11 (core dumped)
I will restore a backup and see how things improve.

PS.: In spite of this I could build and install postfix. I found it somehow did not like openldap support unless I recompiled openldap against openssl 1.0.2 (though it should have been done yesterday already). But it might as well be related to temporarily switching login shell from zsh to csh.

Regards,
Peter
 
OP
OP
P

pvoigt

Member

Reaction score: 5
Messages: 82

I successfully restored a UFS2 dump of my root filesystem. There are first replies on the PR having similar bad experiences.

I am delaying all port upgrades until this issue is solved.

Regards,
Peter
 

xtaz

Well-Known Member

Reaction score: 112
Messages: 416

I agree. Something is very wrong. I upgraded to 1.0.2 and then found the load average on my box had gone up to 50+ with hundreds of PHP-FPM processes. Recompiling things doesn't seem to have made any difference. Various things are hanging or racing. I've downgraded back to 1.0.1 to fix it all.
 
OP
OP
P

pvoigt

Member

Reaction score: 5
Messages: 82

I agree. Something is very wrong. I upgraded to 1.0.2 and then found the load average on my box had gone up to 50+ with hundreds of PHP-FPM processes. Recompiling things doesn't seem to have made any difference. Various things are hanging or racing. I've downgraded back to 1.0.1 to fix it all.
Yeah, I thought about downgrading as well. First I thought postfix is faulty. But as time proceeded more and more services core dumped and it turned out that obviously openssl is broken. And to be honest: I have no idea how I could have managed downgrading openssl. I don't know where to obtain from an older source package and how to install it. Could you please briefly describe your downgrading steps.

Regards,
Peter
 

xtaz

Well-Known Member

Reaction score: 112
Messages: 416

My ports tree is checked out using subversion so all I had to do to downgrade was find out the last revision number with svn log /usr/ports/security/openssl | more and then svn up -r381697 /usr/ports/security/openssl followed by a reinstall. I might do some more testing on this today now I have some time. It occured to me afterwards that I had tried recompiling things like lang/php56 and security/php56-openssl but that's not going to the the source of the problem I had with hundreds of PHP-FPM processes. It's more likely to be in www/nginx which I didn't try and recompile. So I'll do some more testing later on.

I run mail/postfix as well so I can see what happens when I try recompiling that seeing as this was the original purpose of this thread.
 

xtaz

Well-Known Member

Reaction score: 112
Messages: 416

Well I've retested this. And there's still some really strange problems. I still get the race condition with PHP-FPM taking huge amount of processes and load average shoots up to 50+. And I just tried to run a /bin/sh shell script and it said this:
Code:
sh: environment corrupt; missing value for USERNAME
Backing out to the old openssl fixes everything. But postfix works fine.....
 
OP
OP
P

pvoigt

Member

Reaction score: 5
Messages: 82

Well I've retested this. And there's still some really strange problems. I still get the race condition with PHP-FPM taking huge amount of processes and load average shoots up to 50+. And I just tried to run a /bin/sh shell script and it said this:

Code:
sh: environment corrupt; missing value for USERNAME
Backing out to the old openssl fixes everything. But postfix works fine.....
I can confirm this /bin/sh error which appeared on my system with /usr/local/bin/zsh as well. And the initial postfix related error was environment related. From this I conclude that the initial postfix build error was a consequence of the corrupt environment only.
 

xtaz

Well-Known Member

Reaction score: 112
Messages: 416

So having 1.0.2 installed is corrupting the environment in some fashion? That might explain why PHP-FPM goes mental as that spawns new processes on every request and works with environment variables. I don't get why only a few people seem to be having problems and most people seem happy with it. There must be something in common with people that are affected. I run static shells/bash as a root shell, UTF-8 as a locale, all my terms are run within sysutils/tmux, amd64. Not really changed much else other than setting various env variables.
 
OP
OP
P

pvoigt

Member

Reaction score: 5
Messages: 82

So having 1.0.2 installed is corrupting the environment in some fashion? That might explain why PHP-FPM goes mental as that spawns new processes on every request and works with environment variables. I don't get why only a few people seem to be having problems and most people seem happy with it. There must be something in common with people that are affected. I run static shells/bash as a root shell, UTF-8 as a locale, all my terms are run within sysutils/tmux, amd64. Not really changed much else other than setting various env variables.
Did you try to rebuild openssl with ASM=off as talsamon suggested?
 

xtaz

Well-Known Member

Reaction score: 112
Messages: 416

Did you try to rebuild openssl with ASM=off as talsamon suggested?
Interesting. I forgot about trying that. You know what? That actually appears to have solved the problem. Everything is working now. So I guess it's actually something hardware specific. It's an Intel Atom D525. Oh well, thanks guys!
 
OP
OP
P

pvoigt

Member

Reaction score: 5
Messages: 82

Interesting. I forgot about trying that. You know what? That actually appears to have solved the problem. Everything is working now. So I guess it's actually something hardware specific. It's an Intel Atom D525. Oh well, thanks guys!
Well, I myself did not yet find time enough to rebuild with talsamon suggested build option but it has been since on my todo list. It's good to hear that it seems to work :) I need some more time for my next tests because I feel not very safe with a possibly required downgrade of OpenSSL. In one of your recent posts you have briefly described how to downgrade using svn. I am not quite sure, if after checking out the specified revision with
svn up -r381697 /usr/ports/security/openssl 1.) the ports tree index is updated as well and 2.) if I can use portmaster/pkg or if make install is subsequently required.

Regards,
Peter
 

xtaz

Well-Known Member

Reaction score: 112
Messages: 416

I've been updating the bugzilla PR for this. Just given them build logs for ASM on and off this morning. I'm now building lang/gcc and I'm going to try to compile security/openssl with ASM enabled using gcc to see if that makes any difference as by default it will be compiled using clang on 10.1.

Regarding downgrading. It depends if you have your ports tree checked out using devel/subversion or not. I personally use devel/subversion so I could use that command to check out an earlier commit, then I did make deinstall install clean to remove 1.0.2 and install 1.0.1. However it would be worth just trying it with ASM switched off like I did rather than downgrade. This would not have updated the index either, and I don't think you can do downgrades using pkg.
 

Junkie

Member

Reaction score: 7
Messages: 37

I had the same problem with minidlna
Code:
root@daemon # minidlnad -R -u dlna -f /usr/local/etc/minidlna.conf
sh: environment corrupt; missing value for BLOCKSIZ?
sh: environment corrupt; missing value for BLOCKSIZ?
sh: environment corrupt; missing value for BLOCKSIZ?
sh: environment corrupt; missing value for BLOCKSIZ?
The problem was fixed after OpenSSL rebuild without ASM support. Thanks talsamon.
 

xtaz

Well-Known Member

Reaction score: 112
Messages: 416

FYI. I recompiled it using gcc48 with ASM enabled and the same fault happens. So it's not a clang specific issue.
 

spil

New Member

Reaction score: 1
Messages: 8

There are 2 distinct problems addressed in this thread (and the PR). This post describes conflicting OpenSSL libraries from base and ports (/usr/lib/libssl.so.7 vs /usr/local/lib/libssl.so.8 and the same for libcrypto). This didn't surface before on 10 as base and ports both had shlib version .7 with OpenSSL 1.0.1. With 1.0.2 the SHLIBVER was bumped leading to problems.

ANYONE using OpenSSL 1.0.2 will have to compile ANYTHING that relies on OpenSSL with the following in their
/etc/make.conf:
Code:
WITH_OPENSSL_PORT=   yes
#OPENSSL_PORT= security/openssl
(specifying OPENSSL_PORT is not required, allows you to select either OpenSSL or LibreSSL)
This tends to equate to rebuilding a larger number of ports.

Notable dependency examples that will cause this issue are:
When reporting errors, please make sure you check the output of readelf -d on binaries and libraries as well as output from ldd(1). In general you will notice the conflicting shlib-version in the ldd(1) output as that show recursive dependencies whereas readelf -d will only show the direct dependencies of the binary/library.

Check https://bugs.freebsd.org/195796 Bug 195796 - exp-build with WITH_OPENSSL_PORT=yes and SSLv2/SSLv3 disabled for lists of ports that are not linking OpenSSL properly (yet) like virtualbox-ose (See PR 199377 for a partial fix).
 

kpa

Beastie's Twin

Reaction score: 1,807
Messages: 6,318

The WITH_OPENSSL_PORT setting is not enough, some ports will still link against the base system OpenSSL libraries trough for example the GSSAPI libraries.

You can compile ftp/curl with GSSAPI_NONE if you don't need GSSAPI support in it. This will stop it from linking against the base system OpenSSL libraries. I didn't find similar workarounds for other ports though.
 

spil

New Member

Reaction score: 1
Messages: 8

The WITH_OPENSSL_PORT setting is not enough, some ports will still link against the base system OpenSSL libraries trough for example the GSSAPI libraries.

You can compile ftp/curl with GSSAPI_NONE if you don't need GSSAPI support in it. This will stop it from linking against the base system OpenSSL libraries. I didn't find similar workarounds for other ports though.
I do believe I fixed that recently with the latest curl patch...
Code:
IGNORE=  with GSSAPI_BASE, configure failed to detect OpenSSL/LibreSSL from ports and link against base OpenSSL
 
Top