crontab and rc.d profile

[Mage]

It seems to me that cron and rc.d scripts ignore profile files.

On my servers and desktop I use Apache with RVM and Passenger and Rails. Apache is installed from ports and the rest is installed with my user into the home. This setup requires several lines to be added to the .bashrc and .bash_profile as I use bash.

When I run /usr/local/etc/rc.d/apache22 restart as root from command line then everything works fine.

When I reboot the system apache displays error 500 pages because it cannot see env variables and so required to run RVM properly. It can be fixed by simply restarting apache. I think that the normal user's .bash_profile only runs when I run /usr/local/etc/rc.d/apache22 restart from command line as root but not when the system runs it.

(Apache runs Passenger and RVM with my user this is why it has to be mine .bash_profile file.)

Another related thing is that I have these lines in /etc/profile:

export LANG=en_US.UTF-8
export LC_ALL=en_US.UTF-8

There is shell script in my non-root home folder has to be run several times per day. (Spamassassin autowhitelist).

The result ran by cron differs from the result ran by command line. The difference is that grep cannot see characters like "ü" from cron. It seems to me that cron doesn't use /etc/profile.

How can I solve these issues in an elegant way? I would like to make cron and rc.d scripts work exactly same ran either by system or by the user if this isn't a security risk.

A side note that I tend to delete /.cshrc and /.profile after installing FreeBSD because I don't like to have files in /.

 

[Mage]

I also would like to make sure that apache knows that LANG is en_US.UTF-8 even after reboot.

 

[SirDice]

Don't change root's shell, leave it set to /bin/csh. You'll have problems next time you upgrade.

For Apache see post #4 in this thread: Thread 7221.

 

[Mage]

Don't change root's shell, leave it set to /bin/csh. You'll have problems next time you upgrade.

I prefer colored prompt. I see the color and I know that I am on my server and I am root. It feels more safe. Comes from Gentoo.

Also I set toor user's password to be able to login without bash.

However I tried to switch both root's and my user's shell to csh and I still have the problem that cron scripts run with different locale. Namely they seem to ignore /etc/profile which should be a bash/csh/zsh independent file.

I would rather fix the root of the problem than fix every script and application individually.

Supposedly there should be a way to run cron and rc.d scripts with same environment the user would do. I wouldn't like to have any difference between running /usr/local/etc/rc.d/* as root or by the system startup because it can cause several hidden errors like different locale. These hidden things can turn out later and cause pain.

For example most email addresses don't have unicode characters. Testing didn't show the issue. But we happen to have an IDN address and I was just lucky to find the error. It could stay there for years.

Anyway, how can I make cron and startup scripts use the profile files?

 

[Mage]

Or should I put the locale settings to somewhere else? To /etc/rc.conf or even /etc/rc?

Even then I still would like to set up that cron runs the user's profile.

 

[kpa]

Why don't you set up the job to be run from the user's crontab(5)? Even if the user has logins disabled you edit the crontab of that user as root:

# crontab -u www -e

Also look at login.conf(5) for setting locale(1) and other settings independently of the used shell.

 

[Mage]

Why don't you set up the job to be run from the user's crontab(5)?

I always do that (crontab -e as the user).

Also look at login.conf(5) for setting locale(1) and other settings independently of the used shell.

Thank you.

I just commented out the:

export LANG=en_US.UTF-8
export LC_ALL=en_US.UTF-8

lines in /etc/profile and changed /etc/login.conf to:

default:\
        :passwd_format=md5:\
        :copyright=/etc/COPYRIGHT:\
...
        :umask=022:\
        :charset=UTF-8:\
        :lang=en_US.UTF-8:\
        :setenv=LC_ALL=en_US.UTF-8:

then ran cap_mkdf and reboot.

Unfortunately it got worse than setting these in /etc/profile.

usr/home/mage $ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_ALL=

As you see that LC_ALL doesn't have any value. I don't know why. I double-checked everything.

Now look this:

mage@martina /usr/home/mage $ cat bin/locale1.sh 
#!/usr/local/bin/bash
/usr/bin/locale >> /home/mage/locale1.txt

mage@martina /usr/home/mage $ crontab -e
* * * * *               /home/mage/bin/locale1.sh

mage@martina /usr/home/mage $ cat locale1.txt 
LANG=
LC_CTYPE="C"
LC_COLLATE="C"
LC_TIME="C"
LC_NUMERIC="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_ALL=

Locale C has no benefit for me. I know that it is God's curse since Babylon. I understand that people on this Earth should never have a common language. I just hope that at least we should try to have a common character encoding. Which definiately has to be UTF-8. I wish I could burn LANG=UTF8 into the boot loader code. But I can't.

 

[Mage]

As for Apache I inserted into /usr/local/etc/rc.d/apache22

PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/root/bin

I don't like it and I still would like to make rc scripts ran by the system on boot and ran by root work the same way, however this modification works for my issue.

 

[kpa]

No need to change the global /etc/login.conf for just testing the settings, there's an example .login_conf in your home directory

# $FreeBSD: src/share/skel/dot.login_conf,v 1.3.36.1.4.1 2010/06/14 02:09:06 kensmith Exp $
#
# see login.conf(5)
#
#me:\
#       :charset=iso-8859-1:\
#       :lang=de_DE.ISO8859-1:

Make always sure there's newline after the last line, cap_mkdb(1) will fail silently if there's no newline.

Remove the comment signs from the last three lines and change the settings to your liking and run:

$ cap_mkdb .login_conf

 

[Mage]

No need to change the global /etc/login.conf for just testing the settings, there's an example .login_conf in your home directory.
...

Thank you. The results are the same.

$ cat .login_conf
# $FreeBSD: releng/9.0/share/skel/dot.login_conf 77995 2001-06-10 17:08:53Z ache $
#
# see login.conf(5)
#
me:\
        :charset=UTF-8:\
        :lang=en_GB.UTF-8:

I used en_GB to see that the file is correctly processed.

$ locale
LANG=en_GB.UTF-8
LC_CTYPE="en_GB.UTF-8"
LC_COLLATE="en_GB.UTF-8"
LC_TIME="en_GB.UTF-8"
LC_NUMERIC="en_GB.UTF-8"
LC_MONETARY="en_GB.UTF-8"
LC_MESSAGES="en_GB.UTF-8"
LC_ALL=

Seems fine except that LC_ALL has no value. However it is still not good for cron.

$ rm locale1.txt

[wait and check system date until "00" seconds...]

$ cat locale1.txt 
LANG=
LC_CTYPE="C"
LC_COLLATE="C"
LC_TIME="C"
LC_NUMERIC="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_ALL=

 

[lbol]

/etc/profile like other startup files will only be read by bash if invoked as a login shell. See section INVOCATION in the bash man page

 

[kpa]

Yeah, crontab(5) entries are run in non-login shell and it's probably easiest to just set LC_ALL directly in the crontab file.

LC_ALL=en_GB.UTF-8

*/30 * * * * locale

 

[Mage]

/etc/profile like other startup files will only be read by bash if invoked as a login shell. See section INVOCATION in the bash man page

I see. Thank you.

So how can I make FreeBSD really system-wide UTF-8? The methods I tried don't work. Even the manual http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/using-localization.html says editing profile files and login.conf, however it is not sufficient as you see.

If I don't want anything *ever* in any case by accident run with locale=C on my server where should I put this information?

Otherwise it will cause nothing but errors in a "unicode wannabe" environment.

Please tell me that I can do it. I just donated to the FreeBSD fund so I wouldn't want to change operating system for awhile.

 

[Mage]

Yeah, crontab(5) entries are run in non-login shell and it's probably easiest to just set LC_ALL directly in the crontab file.

The thing I would like to avoid is to set it at 10 different place and fail at 11th.

What if I put it to rc.conf or /etc/rc? I really want it to be system-wide.

 

[kpa]

You don't want to change locale for root from the default "C", there are documented cases where building of ports fail with non standard locales. Also I'm not very convinced that all the system shell scripts are yet UTF-8 safe.

 

[Mage]

You don't want to change locale for root from the default "C", there are documented cases where building of ports fail with non standard locales. Also I'm not very convinced that all the system shell scripts are yet UTF-8 safe.

If I have to choose I would prefer failing at building a port instead of losing or corrupting user data. Failing at port building is something I see instantly. And none of the ports I use failed so far while I use UTF-8 as root.

Having "C" locale without knowing it causes hidden but serious errors. For example regular expressions will work differently. You won't see any error because they'll run fine but the result will be different. For example they won't recognize internationalized characters as alphanumeric. And your users will have very annoying issues with string validation and sorting. Without your applications ever throwing an exception. It is as bad as silent data corruption.

It already happened to me when an IDN email didn't get through a regular expression. I was lucky enough to compare the results from command line and crontab. Only one line had difference. If we receive it later maybe I will never notice.

This is the issue you want to avoid at multi-language websites. It is a 100% sure source of random errors in the future. Like having a single disk storage. It will fail for 100%. The only questions is that when will that happen. If you know that it will happen for sure, wouldn't you like to avoid?

Sooner or later you will forget to add export LANG=... to some of your crontab scripts. Or somewhere else where you shouldn't add to because you think you already set it. And when you run your tests it works fine.

It is sad that UTF-8 is still an issue in 2012. I am not bashing FreeBSD becuse it is a global problem. However I believe that many people setup FreeBSD for webservers like me. I already spent some time to figure out that PostgreSQL has hardcoded LC_CTYPE=C in it's FreeBSD rc.d initfile even when compiled with unicode (ICU) support. Also its unicode support didn't work with server version 9.1 some months ago with the very same options it worked fine with 9.0. And it didn't even tell that. It "only" made wrong string ordering. Which can cause much more serious problems than seeing lines in wrong order. It can even cause silently losing records as the btree index also might not work correctly. We had these issues in real already.

As a developer I feel spending too much time to forge systems to be real UTF-8 and not just "almost UTF-8". As UTF-8 is backward compatible to LATIN1 I also think this shouldn't be still an issue nowadays.

And I mean no offense and I see that you all try to help. Considering that I take the risk of failing with a port building where should I put that language info for all users and all cases?