HOWTO: Bareos Backup Server on FreeBSD

vermaden

Son of Beastie

Thanks: 1,011
Messages: 2,673

#1
I would like to share new ๐—•๐—ฎ๐—ฟ๐—ฒ๐—ผ๐˜€ ๐—•๐—ฎ๐—ฐ๐—ธ๐˜‚๐—ฝ ๐—ฆ๐—ฒ๐—ฟ๐˜ƒ๐—ฒ๐—ฟ ๐—ผ๐—ป ๐—™๐—ฟ๐—ฒ๐—ฒ๐—•๐—ฆ๐—— post about installation and configuration of open, free and complete backup solution.

https://vermaden.wordpress.com/2018/05/01/bareos-backup-server-on-freebsd/

#verblog #bareos #freebsd #postgresql #backup #restore #recovery
 

gkontos

Daemon

Thanks: 457
Messages: 2,099

#2
Bareos is a good open source alternative forked from Bacula. But:

  • Their RDMBS design sucks. They are trying to fix that in the new version. Still the new version has a lot of bugs and I don't think it is production ready yet.
  • Using the WEB UI for restoring files on large environments simply does not work. You have to use the bconsole.

We are currently using the director on a CentOS box. The database sits on a 96GB RAM SSD 2X Processors CentOS box. And we have 10 FreeBSD ZFS Storages running the SD. We backup around 170 servers.
 
OP
OP
vermaden

vermaden

Son of Beastie

Thanks: 1,011
Messages: 2,673

#3
@ gkontos
Thanks for sharing your thoughts. I have several questions if I may.


  • Their RDMBS design sucks. They are trying to fix that in the new version. Still the new version has a lot of bugs and I don't think it is production ready yet.
Sucks in what way?

You mean that 17.x is bugged currently? Even in 17.2.5 maintenance version? What problems do you hit?

  • Using the WEB UI for restoring files on large environments simply does not work. You have to use the bconsole.
You say about 16.x or 17.x version here? Does not work you mean hangs in the process or something else?
 

gkontos

Daemon

Thanks: 457
Messages: 2,099

#4
@ gkontos
Thanks for sharing your thoughts. I have several questions if I may.


Sucks in what way?
The database has a size of 240GB. 80% of that is being used by one table. File. We are using Percona and not Postgresql but I think that does not change much.

You mean that 17.x is bugged currently? Even in 17.2.5 maintenance version? What problems do you hit?
I am still using the 16.X version. The upgrade path is complicated and there is no roll back. However, I am following their mailing lists and I see that there are many problems.

You say about 16.x or 17.x version here? Does not work you mean hangs in the process or something else?
Yes, the 16.X version always, however they say that the same problem occurs in the 17.X version. My environment is very demanding and we are backing up managed clients. Some of them are running very large shared hosting WHM servers with really a lot of data.
When you try to restore from the Web Ui and you choose "Merge all related jobs to last full backup of selected backup job" the application just hangs there. This is a know problem and it has been discussed a lot in their lists.

A temporary solution that I have found since my database server has enough horsepower, is to run the following cron job in the director:

*/2 * * * echo ".bvfs_update"|/usr/sbin/bconsole

Still, there are some cases that I need to use the bconsole. Which is fine by me but the purpose is to have other people to be able to use the restore option when I am not around!
 

VladiBG

Active Member

Thanks: 82
Messages: 235

#6
When you disable the sendmail in rc.conf you may also consider to disable submission and queue runner.

sendmail_enable="NONE"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"


Then when there's no sendmail running you don't need the periodic to clear the hoststat so you can disable it with
daily_clean_hoststat_enable="NO"
 
OP
OP
vermaden

vermaden

Son of Beastie

Thanks: 1,011
Messages: 2,673

#7
@ VladiBG

Its not that simple :)

Generally when You set sendmail_enable to NONE it will make all other to NO (including sendmail_enable), check /etc/rc.d/sendmail for details.

This is on a 11.1-RELEASE-p9 system.
Bash:
% grep -A 6 'case ${sendmail_enable} in' /etc/rc.d/sendmail
case ${sendmail_enable} in
[Nn][Oo][Nn][Ee])
        sendmail_enable="NO"
        sendmail_submit_enable="NO"
        sendmail_outbound_enable="NO"
        sendmail_msp_queue_enable="NO"
        ;;
... and this sendmail_enable=NONE was set by FreeBSD Installer (bsdinstall) ... and yet in man rc.sendmail page it is said that NONE option is deprecated and should not be used. It will be removed in a future release ...

Code:
sendmail_enable
     (str) If set to ``YES'', run the sendmail(8) daemon at system
     boot time. If setto ``NO'', do not run asendmail(8) daemon to
     listen forincoming network mail.This does not preclude a
     sendmail(8) daemonlistening on the SMTP port of the loopback
     interface. The ``NONE'' option is deprecated and should not be
     used.  It will be removed in a future release.
About /etc/periodic.conf there are a lot of things that can be disabled, but as asticle was too big already I did not wanted to cover another 'case' for FreeBSD setup as it was Bareos focused, but Yes, that also can be added.
 

dasti

New Member


Messages: 12

#8
Bareos is a good open source alternative forked from Bacula. But:

  • Their RDMBS design sucks. They are trying to fix that in the new version. Still the new version has a lot of bugs and I don't think it is production ready yet.
  • Using the WEB UI for restoring files on large environments simply does not work. You have to use the bconsole.

We are currently using the director on a CentOS box. The database sits on a 96GB RAM SSD 2X Processors CentOS box. And we have 10 FreeBSD ZFS Storages running the SD. We backup around 170 servers.
thanks for sharing your experience on bareos, do you have any experience on bareos vs bacula / pros and cons ?
 

sidetone

Aspiring Daemon

Thanks: 310
Messages: 900

#9
Generally when You set sendmail_enable to NONE it will make all other to NO (including sendmail_enable), check /etc/rc.d/sendmail for details.

This is on a 11.1-RELEASE-p9 system.
Bash:
% grep -A 6 'case ${sendmail_enable} in' /etc/rc.d/sendmail
case ${sendmail_enable} in
[Nn][Oo][Nn][Ee])
        sendmail_enable="NO"
        sendmail_submit_enable="NO"
        sendmail_outbound_enable="NO"
        sendmail_msp_queue_enable="NO"
        ;;
... and this sendmail_enable=NONE was set by FreeBSD Installer (bsdinstall) ... and yet in man rc.sendmail page it is said that NONE option is deprecated and should not be used. It will be removed in a future release ...

Code:
sendmail_enable
     (str) If set to ``YES'', run the sendmail(8) daemon at system
     boot time. If setto ``NO'', do not run asendmail(8) daemon to
     listen forincoming network mail.This does not preclude a
     sendmail(8) daemonlistening on the SMTP port of the loopback
     interface. The ``NONE'' option is deprecated and should not be
     used.  It will be removed in a future release.
I thought sendmail as a whole was being removed from a future release.
I'm using 11.2, and the NONE option no longer removes all sendmail named startups on reboot. VladiBG and vermaden may have answered that.
 
OP
OP
vermaden

vermaden

Son of Beastie

Thanks: 1,011
Messages: 2,673

#10
I thought sendmail as a whole was being removed from a future release.
I'm using 11.2, and the NONE option no longer removes all sendmail named startups on reboot. VladiBG and vermaden may have answered that.
Disabling sendmail is one thing, taking care of growing /var/spool/clientmqueue is another.

As I disabled sendmail in /etc/rc.conf file.

Code:
# grep sendmail /etc/rc.conf
  sendmail_enable=NONE
  sendmail_submit_enable=NO
  sendmail_outbound_enable=NO
  sendmail_msp_queue_enable=NO
# ps auxwww | grep sendmail | grep -v grep
#
I also took care of that growing problem with separate crontab job.

Code:
# cat /etc/cron.d/sendmail-clean-clientmqueue 
# CLEAN SENDMAIL
*/3     *       *       *       *       root    /bin/rm -r -f /var/spool/clientmqueue/*
 

PMc

Member

Thanks: 27
Messages: 96

#12
The database has a size of 240GB. 80% of that is being used by one table. File. We are using Percona and not Postgresql but I think that does not change much.
And in what way would that suck? I'd suppose thats what a database is built for. BTW: it's 95% in R.17/postgres10.

I am still using the 16.X version. The upgrade path is complicated and there is no roll back.
That may depend on the features used, but I didn't recognize it as complicated, only as time and disk consuming. It will basically need 3 times the diskspace (or 5 times when keeping redolog backup), and there was no more task than executing the conversion SQL commands.
Rollback would basically mean to run a point-in-time recovery on the database to the point before the conversion.

However, I am following their mailing lists and I see that there are many problems.
I tried to read these things back in 2008 when I tried to fix some of the bugs. I was not really comfortable with the working style there. I dont know if or how that may have changed with the bareos fork.
 
OP
OP
vermaden

vermaden

Son of Beastie

Thanks: 1,011
Messages: 2,673

#14
When you disable the queue there's no need to remove the clientmqueue.
Not quite, there will always happen someone/something will just write mail/notification to local account:

Code:
# mail root
Subject: asd
asd
EOT

# ls -l /var/spool/clientmqueue
total 2
-rw-rw----  1 vermaden  smmsp    4 Aug  4 17:58 dfw74FwWGb015949
-rw-rw----  1 vermaden  smmsp  526 Aug  4 17:58 qfw74FwWGb015949
That is why I put this crontab job to not have disk filled with them ...
 

PMc

Member

Thanks: 27
Messages: 96

#15
PMc Are you using Bareos in a production environment or are you just trolling me here?
Hรค?? For production environment you probably should use Microsoft Windows Server and Tivoli Spectrum Protect, and make sure for good MCSE, not to forget ITIL as well. We are hackers.
 

gkontos

Daemon

Thanks: 457
Messages: 2,099

#16
Hรค?? For production environment you probably should use Microsoft Windows Server and Tivoli Spectrum Protect, and make sure for good MCSE, not to forget ITIL as well. We are hackers.
That makes sense now.
 

VladiBG

Active Member

Thanks: 82
Messages: 235

#17
Not quite, there will always happen someone/something will just write mail/notification to local account:

Code:
# mail root
Subject: asd
asd
EOT

# ls -l /var/spool/clientmqueue
total 2
-rw-rw----  1 vermaden  smmsp    4 Aug  4 17:58 dfw74FwWGb015949
-rw-rw----  1 vermaden  smmsp  526 Aug  4 17:58 qfw74FwWGb015949
That is why I put this crontab job to not have disk filled with them ...
change the directory to readonly and disable the queue.
 

sidetone

Aspiring Daemon

Thanks: 310
Messages: 900

#18
change the directory to readonly and disable the queue.
or Soft link it to /tmp, or mount it as a RAM file system. I'm not using a crontab to remove it.

In my opinion, it's better to leave it, so I can see what happened. As long as it doesn't take up over 100MB, all sendmail services are off (not other applications that call it), and doesn't give me warning messages on boot-up, that accomplishes the task well enough for me.

I don't know why they complicated it. Before, sendmail related applications only had to be unset with NONE. Above solutions that removed all related sendmail instances from rc.conf will do.
 
Top