Solved Error Message "cron: putenv: Invalid argument" after 13.0-RELEASE upgrade

dougs

Active Member

Reaction score: 11
Messages: 159

Over the past weekend, two FreeBSD 12.2-RELEASE-p7 servers were upgraded to 13.0-RELEASE using freebsd-update. One of them now sends me an email each time an entry in /etc/crontab is executed as follows:

Code:
cron: putenv: Invalid argument

The other server does not exhibit such behavior. I googled for anything that would shed light on this message but was unable to find anything. I verified the permissions of /etc/crontab as 0644. root:wheel as owner/group. I'm not sure what else I need to do next. Can anyone point me in the right direction?
 

covacat

Well-Known Member

Reaction score: 226
Messages: 474

you probably have a line in /etc/crontab that is malformed and cron thinks is an env var definition

Code:
 [EINVAL]         
                        The function putenv() failed because string is a NULL
                        pointer, string is without an “=” character or “=” is
                        the first character in string.  This does not follow
                        the POSIX specification.
 
OP
dougs

dougs

Active Member

Reaction score: 11
Messages: 159

All of the entries!

The only change as far as I can tell to /etc/crontab during the upgrade was the change from:
Code:
# $FreeBSD: releng/12.2/usr.sbin/cron/cron/crontab 338497 2018-09-06 14:55:54Z brd $
to
Code:
# $FreeBSD$

Oddly enough, the other server has the exact same change and is not exhibiting odd behavior.

I'll go through each line and see if I can identify anything odd.
 
OP
dougs

dougs

Active Member

Reaction score: 11
Messages: 159

This is the /etc/crontab file:
Code:
[root@artemis 14.Jun 10:56am ~]# vi /etc/crontab
# /etc/crontab - root's crontab for FreeBSD
#
# $FreeBSD$
#
SHELL=/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin
MAILTO=admin@example.com
=======

#
#minute hour    mday    month   wday    who     command
#
# Save some entropy so that /dev/random can re-seed on boot.
*/11    *       *       *       *       operator        /usr/libexec/save-entropy
#
# Rotate log files every hour, if necessary.
0       *       *       *       *       root    newsyslog
#
# Perform daily/weekly/monthly maintenance.
1       3       *       *       *       root    periodic daily
15      4       *       *       6       root    periodic weekly
30      5       1       *       *       root    periodic monthly
#
# Adjust the time zone if the CMOS clock keeps local time, as opposed to
# UTC time.  See adjkerntz(8) for details.
1,31    0-5     *       *       *       root    adjkerntz -a
30      2       *       *       5       root    /root/bin/portupdater.sh cron

# below two lines are the Sandisk SSDs identified as Marvell SSDs. Smartmontools has a bug that was fixed in r4889 (2019-01-11)
#0       3       *       *       5       root    /usr/local/sbin/smartctl -a /dev/ada0
#0       3       *       *       5       root    /usr/local/sbin/smartctl -a /dev/ada1
@daily                                  root    /root/bin/disk_space_Check.sh
@daily                                  root    /usr/sbin/freebsd-update cron
10      7       8-14    *       *       root    [ "$(date '+\%a')" = "Sun" ] && /usr/bin/nice /sbin/zpool scrub zroot
40      7       8-14    *       *       root    [ "$(date '+\%a')" = "Sun" ] && /usr/bin/nice /sbin/zpool scrub zdata
50      7       *       *       *       root    /root/bin/pg_dump_zabbix_to_backup.sh
35      9       *       *       *       root    /root/bin/zfs_health.sh

# zfstools
7       0       *       *       *       root    /usr/local/sbin/zfs-auto-snapshot daily     7
24      0       *       *       7       root    /usr/local/sbin/zfs-auto-snapshot weekly    4
58      0       1       *       *       root    /usr/local/sbin/zfs-auto-snapshot monthly   6
# zxfer
5       2       *       *       *       root    /usr/local/sbin/zxfer -dFkPv -g 376 -T root@backup.example.com -R zdata/pgsql/data zdata/home/install/artemis
25      2       *       *       *       root    /usr/local/sbin/zxfer -dFkPv -g 376 -T root@backup.example.com -R zroot zdata/home/install/artemis
25      2       *       *       *       root    /usr/local/sbin/zxfer -dFkPv -g 376 -T root@backup.example.com -R zroot/ROOT/default zdata/home/install/artemis
25      2       *       *       *       root    /usr/local/sbin/zxfer -dFkPv -g 376 -T root@backup.example.com -R zroot/usr zdata/home/install/artemis
25      2       *       *       *       root    /usr/local/sbin/zxfer -dFkPv -g 376 -T root@backup.example.com -R zroot/var zdata/home/install/artemis
# reduce pgsql archival files after 21 days
0       4       *       *       *       root    find /zdata/pgsql/data/archive -type f -mtime +14d -delete
#0      4       *       *       *       root    /root/bin/walarchivecleanup.sh -p /zdata/pgsql/data/archive -a 14
*       7       *       *       7       root    /usr/local/sbin/zfs-cleanup-snapshots

I am getting the erroneous email every 11 minutes related to /usr/libexec/save-entropy. I also get the same email every hour on the hour from newsyslog. I also received email from the 7:10, 7:40, 7:50 and the 9:35 entries.

See anything odd?
 

Tieks

Well-Known Member

Reaction score: 117
Messages: 284

You might want to try to remove line 8 with the = signs, or at least put a # in front of it.
 

ralphbsz

Son of Beastie

Reaction score: 2,357
Messages: 3,244

That line with the half dozen = signs is left over from a merge. Typically, if a merge fails, the merging tool will put in there:
Code:
<<<<<<
Old version
======
New version
>>>>>>
or something similar. What you have there is wreckage from that process.
 
OP
dougs

dougs

Active Member

Reaction score: 11
Messages: 159

You might want to try to remove line 8 with the = signs, or at least put a # in front of it.
That did it! Oddly, the other server has the exact same line and I do not receive any complaints from it.
 
OP
dougs

dougs

Active Member

Reaction score: 11
Messages: 159

I spoke too soon.

About a hour and half after, I started seeing this via email:

Code:
fsync: open saved-entropy.1: Permission denied

Code:
[root@artemis 14.Jun 5:47pm /usr/libexec]# ll
total 1185
-r-xr-xr-x   1 root  wheel   69240 Jun 13 11:50 atf-check*
...
-r-xr-xr-x   1 root  wheel   10144 Jun 13 11:50 rpc.rusersd*
-r-xr-xr-x   1 root  wheel    9216 Jun 13 11:50 rpc.rwalld*
-r-xr-xr-x   1 root  wheel    7912 Jun 13 11:50 rpc.sprayd*
-r-xr-xr-x   1 root  wheel    4295 Jun 13 11:50 save-entropy*
drwxr-xr-x   2 root  wheel       3 Jun 13 11:50 sendmail/
-r-xr-xr-x   1 root  wheel   39008 Jun 13 11:50 sftp-server*
drwxr-xr-x   2 root  wheel       2 Sep 28  2016 sm.bin/
...

Code:
[root@artemis 14.Jun 5:47pm /usr/libexec]# locate saved-entropy.1
[root@artemis 14.Jun 5:50pm /usr/libexec]#

Not sure what to do.
 

Tieks

Well-Known Member

Reaction score: 117
Messages: 284

Looks like a different problem. Does user operator still have rights to execute /usr/libexec/save-entropy and rights to write to /var/db/entropy? covacat's proposal could help.
 
OP
dougs

dougs

Active Member

Reaction score: 11
Messages: 159

sh -x /usr/libexec/save-entropy
see where / why it fails
Gives me a very long list of output- each line beginning with a plus sign. Appears to be normal. I ran it as root.
 
OP
dougs

dougs

Active Member

Reaction score: 11
Messages: 159

Looks like a different problem. Does user operator still have rights to execute /usr/libexec/save-entropy and rights to write to /var/db/entropy? covacat's proposal could help.

Code:
[root@artemis 15.Jun 6:17am /usr/libexec]# ll | g entropy
-r-xr-xr-x   1 root  wheel    4295 Jun 13 11:50 save-entropy*
[root@artemis 15.Jun 6:17am /usr/libexec]# ll /var/db | g entropy
drwx------    2 operator  operator       10 Jun 14 14:11 entropy/
[root@artemis 15.Jun 6:17am /usr/libexec]# cat /etc/passwd | g operator
operator:*:2:5:System &:/:/usr/sbin/nologin
[root@artemis 15.Jun 6:17am /usr/libexec]# cat /etc/group | g operator
operator:*:5:root
[root@artemis 15.Jun 6:18am /usr/libexec]#

Looks correct, no?
 
OP
dougs

dougs

Active Member

Reaction score: 11
Messages: 159

run it as operator
Code:
[root@artemis 15.Jun 6:22am /usr/libexec]# su operator
This account is currently not available.
[root@artemis 15.Jun 6:24am /usr/libexec]#

How do I run as operator?
 
OP
dougs

dougs

Active Member

Reaction score: 11
Messages: 159

as root
su -m operator
then run it
Gives me a long list of output- each line beginning with a plus sign. Lines seemingly related to save-entropy below:
Code:
+ entropy_dir=/var/db/entropy
+ entropy_save_sz=4096
+ entropy_save_num=8
+ [ ! -d /var/db/entropy ]
+ cd /var/db/entropy
+ [ 1 -gt 8 ]
+ [ 2 -gt 8 ]
+ [ 3 -gt 8 ]
+ [ 4 -gt 8 ]
+ [ 5 -gt 8 ]
+ [ 6 -gt 8 ]
+ [ 7 -gt 8 ]
+ [ 8 -gt 8 ]
+ umask 177
+ n=1
+ [ 1 -le 8 ]
+ save_file=saved-entropy.1
+ [ ! -e saved-entropy.1 -o -f saved-entropy.1 ]
+ break
+ [ -f saved-entropy.1 ]
+ n=2
+ [ 2 -le 8 ]
+ next_file=saved-entropy.2
+ [ -f saved-entropy.2 ]
+ [ saved-entropy.2 -ot saved-entropy.1 ]
+ n=3
+ [ 3 -le 8 ]
+ next_file=saved-entropy.3
+ [ -f saved-entropy.3 ]
+ [ saved-entropy.3 -ot saved-entropy.1 ]
+ n=4
+ [ 4 -le 8 ]
+ next_file=saved-entropy.4
+ [ -f saved-entropy.4 ]
+ [ saved-entropy.4 -ot saved-entropy.1 ]
+ save_file=saved-entropy.4
+ n=5
+ [ 5 -le 8 ]
+ next_file=saved-entropy.5
+ [ -f saved-entropy.5 ]
+ [ saved-entropy.5 -ot saved-entropy.4 ]
+ n=6
+ [ 6 -le 8 ]
+ next_file=saved-entropy.6
+ [ -f saved-entropy.6 ]
+ [ saved-entropy.6 -ot saved-entropy.4 ]
+ n=7
+ [ 7 -le 8 ]
+ next_file=saved-entropy.7
+ [ -f saved-entropy.7 ]
+ [ saved-entropy.7 -ot saved-entropy.4 ]
+ n=8
+ [ 8 -le 8 ]
+ next_file=saved-entropy.8
+ [ -f saved-entropy.8 ]
+ [ saved-entropy.8 -ot saved-entropy.4 ]
+ n=9
+ [ 9 -le 8 ]
+ [ -e saved-entropy.4 -a ! -f saved-entropy.4 ]
+ chmod 600 saved-entropy.4
+ dd 'if=/dev/random' 'of=saved-entropy.4' 'bs=4096' 'count=1'
+ chflags nodump saved-entropy.4
+ fsync saved-entropy.4 .
+ exit 0

Seems to have executed OK.
 

covacat

Well-Known Member

Reaction score: 226
Messages: 474

check if you have any file owned by root in /var/db/entropy

find /var/db/entropy/ \! -user operator
 
OP
dougs

dougs

Active Member

Reaction score: 11
Messages: 159

check if you have any file owned by root in /var/db/entropy
find /var/db/entropy/ \! -user operator
Code:
[root@artemis 15.Jun 6:43am /usr/libexec]# find /var/db/entropy/ \! -user operator
/var/db/entropy/saved-entropy.1
[root@artemis 15.Jun 6:46am /usr/libexec]#
 
Top