Bacula Director Daemon Crashes After MySQL Upgrade

[spomerg]

I hope I posted this in the correct forum; when I did a forum search for "bacula", many of the posts were in this (Web & Network Services) forum. My apologies if I posted in the wrong forum.

I have 3 FreeBSD systems that I backup using 2 other FreeBSD systems running Bacula. My backups have been running successfully for several months now. Last night, instead of receiving my normal 6 (3 systems x 2 backup systems) email messages of cron output from my Bacula jobs, I instead received an error message emailed to me from each Bacula host:

(no debugging symbols found)...ptrace: Operation not permitted.
/var/db/bacula/31553: No such file or directory.
$1 = 0
/usr/local/share/bacula/btraceback.gdb:2: Error in sourced command file:
No symbol table is loaded.  Use the "file" command

When I attempted to connect to my Bacula servers using the Bacula console, I could not connect and I discovered that the Bacula director daemons on each Bacula system weren't running. After starting the Bacula director on each Bacula system with [cmd=]/usr/local/etc/rc.d/bacula-dir start[/cmd], I verified that it was running with [cmd=]ps ax | grep bacula[/cmd], however the director daemon would not be running after about 10 seconds or so.

The only significant, recent change to my 2 Bacula systems is that I did a portupgrade of MySQL from 5.0.91 to 5.0.92 yesterday morning. And since I did this upgrade after the last successful backup, before the problems occurred and the fact that Bacula uses MySQL for it's catalog, I believe that this is suspect.

I did a lot of Googling on the error messages, but could not find anything that helped me enough to solve the problem. I admit I don't really know how to interpret the error message above. I'm also wondering if either the MySQL folks or the Bacula folks might soon have a package upgrade that will fix my problem? Sometimes that has been the case with some ports packages problems in the past.

- Gavin

 

[SirDice]

Is there anything in the MySQL error log that might give a clue?

I don't use Bacula but I'm assuming it has a log file too.

 

[spomerg]

Is there anything in the MySQL error log that might give a clue?

I don't use Bacula but I'm assuming it has a log file too.

Thanks SirDice, you put me on the right track.

I checked to see if MySQL was running and it was not. I checked the MySQL error logs on both Bacula hosts and sure enough they indicated that MySQL wasn't running as of yesterday afternoon:

110110  8:54:32 [Note] /usr/local/libexec/mysqld: ready for connections.
Version: '5.0.91'  socket: '/tmp/mysql.sock'  port: 3306  FreeBSD port: mysql-server-5.0.91
110208 15:44:34 [Note] /usr/local/libexec/mysqld: Normal shutdown

110208 15:44:34  InnoDB: Starting shutdown...
110208 15:44:35  InnoDB: Shutdown completed; log sequence number 0 43665
110208 15:44:35 [Note] /usr/local/libexec/mysqld: Shutdown complete

110208 15:44:35  mysqld ended

110110  8:55:58 [Note] /usr/local/libexec/mysqld: ready for connections.
Version: '5.0.91'  socket: '/tmp/mysql.sock'  port: 3306  FreeBSD port: mysql-server-5.0.91
110208 15:52:31 [Note] /usr/local/libexec/mysqld: Normal shutdown

110208 15:52:31  InnoDB: Starting shutdown...
110208 15:52:31  InnoDB: Shutdown completed; log sequence number 0 43675
110208 15:52:31 [Note] /usr/local/libexec/mysqld: Shutdown complete

110208 15:52:31  mysqld ended

It's likely this was when I did the portupgrades of MySQL. I know from experience that portupgrades of MySQL usually shutdown the daemon. It's for that reason I usually do portupgrades of MySQL like:
[cmd=]portupgrade -R mysql-* ; /usr/local/etc/rc.d/mysql restart ; ps ax | grep mysql[/cmd]
I'm guessing I forgot yesterday for some reason. :r I was tired yesterday and it was near the end of my work day.

After I started mysqld and bacula-dir, I was able to connect with the Bacula console and view other logged messages that indicated the problem:

09-Feb 02:00 entreri-director JobId 0: Fatal error: mysql.c:194 Unable to connect to MySQL server.
Database=bacula User=scott_bacula
MySQL connect failed either server not running or your authorization is incorrect.
09-Feb 02:00 entreri-director: Fatal Error at mysql.c:368 because:
rwl_writelock failure. stat=22: ERR=Invalid argument

Problem solved. Thanks again for pointing out the obvious to me, SirDice. (CHECK LOGS, DUH!) :e