Porting zimbra

[Mel_Flynn]

I'm going to take a stab at porting zimbra to FreeBSD. No promises yet, as they've not exactly made it easy, by requiring specific versions of software and taking these versions into their build process. So I'm gonna have to take it apart and see where things fail when you're trying to use actual ports, rather then their versions.
What I've seen on the web is one old patchset (http://www.zimbra.com/forums/developers/5494-freebsd-patchset-zimbra-released.html) (no offense Doug), but I may have missed something or maybe I'm doing something somebody is already trying or has tried and failed.
In that case I'd like to know / compare notes.

[gvanessen]

If Zimbra was ported to FreeBSD their would be a lot of happy users - see petition and comments on this page (http://www.freebsdnews.net/2008/03/24/zimbra-freebsd-petition/).

Matt Olander from iXsystems may be able to help you or give you some hints - see post on PC-BSD forum (http://forums.pcbsd.org/viewtopic.php?t=6110).

[Mel_Flynn]

Their build system really inhales fluids. For this to become a real port (respecting $PREFIX), I'm gonna have to patch a lot (think thousands) of hardcoded "/opt/zimbra" references. On the plus side, once this is done, Yahoo might actually see the benefits of not having a hardcoded installer path, but one that respects compile time options.
Matt saw that and decided to just work with the /opt/zimbra path. For a real port that's unacceptable. And I really want a real port, cause Zimbra's main weakness are the hardcoded 3rd party software versions that trail weeks/months behind security patches.
On the plus side, Common/IM and Server compile ok and I've pretty much discected what their installer is trying to accomplish and started working on converting that to sensible BSD makefiles.

[MarS]

A FreeBSD port for Zimbra? That would be a dream coming true. :)

[solko-fbsd]

Porting (that is using /usr/ports for whole ThirdParty) zimbra to FreeBSD is a foolish idea. Even linux doesn't have a true port. Zimbra has a reason for not doing so. Using known versions of third party software and building in a pre-set enviroment greatly simplifies support/development.
I broke my wow not to ever install linux just few days ago. All this 'cos of zimbra! After a nightmare weekend and some google love, I have a happy working FreeBSD zimbra installation (instead of the unlucky linux).
Since I am afraid of sorting out all the licensing issues, I can't provide the packages publicly :( unless someone helps me, however I can provide you with fairly straightforward instructions and the altered zimbra source. There are still some bugs to catch but all of them are purely cosmetical (in the end, we would like one "make install", don't we? and we would also very much like most of the ThirdParty installed from ports. not counting rpm->pkg_add).

Zimbra sources are totally linux specific and almost everything is hardcoded.. so I totally hardcoded it into FreeBSD specific :-). Currently only FreeBSD7/amd64 but i386 is really only a manual work issue.
Since we can't make a real port, we should at least try to provide ports like:
- zimbra5-build-enviroment,
- zimbra5-install-enviroment.
That should simplify it a lot as I ended up with 140 installed packages after building every prerequisite from ports (to build zimbra). Once that is done.. we are free to try that foolish porting ;).

Testers wanted! Send me a message if You are interested.

[phoenix]

Porting (that is using /usr/ports for whole ThirdParty) zimbra to FreeBSD is a foolish idea.

Not really. Breaking up the zimbra port into its constituent parts would be foolish (ie mail/postfix, databases/mysql, www/jetty, and so on, with mail/zimbra being a meta-port). But creating a mega-port (note the g) that took the whole zimbra source tree, patched it to work with FreeBSD, compiled it, and installed it to /usr/local/zimbra, with the appropriate CONFLICTS line, should be doable. You could even install it to /opt, to keep the number of "remove hardcoded path" patches close to 0.

Since I am afraid of sorting out all the licensing issues, I can't provide the packages publicly

Last time I checked, there was no restriction on packaging the OSS version.

Zimbra sources are totally linux specific and almost everything is hardcoded.. so I totally hardcoded it into FreeBSD specific :-).

You'd think, with all the work done on creating the Solaris port, and now the Windows port, that there wouldn't be that many Linux-isms left. But that could be wishful thinking. :)

Since we can't make a real port,

Why not? See above.

[solko-fbsd]

Not really. Breaking up the zimbra port into its constituent parts would be foolish (ie mail/postfix, databases/mysql, www/jetty, and so on, with mail/zimbra being a meta-port). But creating a mega-port (note the g) that took the whole zimbra source tree, patched it to work with FreeBSD, compiled it, and installed it to /usr/local/zimbra, with the appropriate CONFLICTS line, should be doable. You could even install it to /opt, to keep the number of "remove hardcoded path" patches close to 0.
Well, that's what I had in mind. Let's not break it up.


Last time I checked, there was no restriction on packaging the OSS version.
You need to include prepackaged diablo-jdk15/diablo-jdk16 in the zcs package. And I am not quite sure if that's okay.


You'd think, with all the work done on creating the Solaris port, and now the Windows port, that there wouldn't be that many Linux-isms left. But that could be wishful thinking. :)


Sadly, the opensource version you can download via Perforce is completely linux specific with hacks to allow for a Mac build/install. But since FreeBSD port does not need to care about linux and since we can make a port once every major zcs release.. that's really just an annoyance instead of a real problem.
http://wiki.zimbra.com/index.php?title=Building_Zimbra_using_Perforce#Bui lding_Zimbra_Collaboration_Suite:_.22FRANKLIN.22

Why not? See above.
Not a true port.. only a mega-port :-). But I guess there is no difference for the end-user.

I Should be able to provide instructions/packages in few days. In the meantime, I will try to make the building process as seamless as possible.

[solko-fbsd]

So.. zimbra is working as expected:
- all services up and running,
- i can send/receive emails,
- i can add/del users,
- i can log in with users.

I am now in the state of patching all those bash/perl scripts and fixing configuration errors (you would think there can't be any, wouldn't you?).
Startup of java-stuff is insanely slow and some things are kinda irracional (like altering postfix configuration at startup via postconf -e while postfix not running..).
BUT. I started with zimbra 4 days ago so I need to learn how it works first.

[phoenix]

You need to include prepackaged diablo-jdk15/diablo-jdk16 in the zcs package. And I am not quite sure if that's okay.

Hrm, couldn't you just add a DEPENDS line on the diable-jdk, and let the user install that version? Or is this needed internally by Zimbra, and needed during the compilation/installation process?

Sadly, the opensource version you can download via Perforce is completely linux specific with hacks to allow for a Mac build/install.

That sucks. Maybe we should put in an RFE for FreeBSD support, or to make the code more portable. I know we (as paying Zimbra customers) would really like to be able to put Zimbra onto a box with ZFS support. :)

We're currently looking at using a FreeBSD box for the backend, using ZFS and iSCSI, with Ubuntu Server systems as the frontend, booting off CompactFlash and using iSCSI for storage.

Not having to use Ubuntu anywhere would be even better. :)

I Should be able to provide instructions/packages in few days. In the meantime, I will try to make the building process as seamless as possible.

Good luck, and keep up the good work!!!

[solko-fbsd]

Zimbra build/packaging process includes jdk in the core package so that it gets installed first. Some packages (core,mta,apache,spell,proxy.. etc) use java as pre/postinstall package procedure so it needs to be present. However, if we take a different approach than installer and fully hack it to support FreeBSD only, we can ask user to install jdk prior to installing zcs (maybe you can distribute binary jdk but you can't distribute sources).

Easter is screwing my worktime :(, I will post the binary packages tomorrow so that someone can catch the bugs I missed. I am not aware of all the features so I can't test them out. All the ThirdParty should be okay but some of those scripts are really silly and I am thinking of rewriting them.
This is FreeBSD after all and we don't like sloppy stuff :-) (no offense zimbra team!).

Screw the license issues, hopefully noone gonna sue me..

[solko-fbsd]

As promised I just built a package for you:
http://mango.euba.sk/zimbra/zcs-install-FreeBSD_amd64.tar.gz

For those who would like to compile it on i386 (source+output from build -> package):
http://mango.euba.sk/zimbra/zcs-source_and_binaries-FreeBSD_amd64.tar.gz

Please follow the included README.FIRST.

I won't be around during Easter so good luck testing! :-)

[solko-fbsd]

Forgot to include ldconfig in the install script.. should be fixed now. If you encounter problems, please try:

ldconfig -m /opt/zimbra/lib
ldconfig -m /opt/zimbra/sleepycat/lib
ldconfig -m /opt/zimbra/openldap/lib
ldconfig -m /opt/zimbra/cyrus-sasl/lib

[MarS]

Tried it -- installation was successful, can login (Webclient and Admin Console) but didn't have enough time to test the actual functionality yet. Keep up the good work! (Now we just need to convince Zimbra to officially provide ZCS for FreeBSD... :P)

[solko-fbsd]

Yay! Glad to hear that. However, that's the only feedback I received so far. I can't fix stuff I don't know about. So I need You (plural) to tell me what to fix! Once that's done, I will send zimbra team my patch towards the actual stable version.

TODO so far:
a) make perl deps from /usr/ports
b) make clamav from /usr/ports 'cos there is a bug every month
c) build FreeBSD package
d) do it for i386
e) remake 600.daily script into /etc/newsyslog.conf, original script rotates log files without notifying daemons so they stop logging..

If i386 package is all that you need, say so. I don't run any recent i386 but I can install some virtual machine.

[phoenix]

Nice. I'll have to see if I can create a VM somewhere to test this.

We're piloting Zimbra NE on Ubuntu Server right now for staff (need the Outlook Connector and Mobile Sync features). If that works, we're considering using Zimbra OS for students. Maybe we'll put that on FreeBSD. :)

[solko-fbsd]

I am moving to:
http://www.zimbra.com/forums/installation/28723-zcs-5-0-16-freebsd_amd64.html#post134904

[iceburn]

I would like the i386 version :)

Great work with the port! Keep it coming!

Hopefully we can get this into the ports tree, even if it does install into /opt/zimbra, atleast its in there for starters.

[smooth]

amazing job really thanks allot

[solko-fbsd]

There was no reply/feedback whatsoever in past month so I thought people did not really care and I stopped spending several hours/day by migrating zimbra to FreeBSD-ports.

I am using zimbra on several FreeBSD servers at the moment and I found out that the binary packages I provided don't really work that great out of the box (as I thought).. so I assumed lack of interest as noone ever asked me a question.

I will continue my efforts then.

[phoenix]

At least for us, it's not lack of interest so much as lack of time for testing. :( We're quite interested in any work toward a port of Zimbra OSS to FreeBSD. Then we can run it on local storage using ZFS, if need be. :)

Keep up the good work, don't be discouraged. :D

[iceburn]

Keep up the good work!

I'll be testing the AMD64 package soon (hopefully this weekend), I need to free up some more space as its heavily used.

It'll be installed in a jail and I'll write up a summary of what I did to get it going.


I'll try to compile the i386 after that, it was quite a download 2G! :O

[iceburn]

http://zimbra.ijichi.org/documentation/freebsd-port

Looks like someone else is trying to do it as well.

[mousaka]

It'll be installed in a jail and I'll write up a summary of what I did to get it going.
I'm very interested in your summary, hopefully with a successful result.

I didn't manage to get a working Zimbra inside a jail so far. Sadly I was only able to spend little time on that topic.

mousaka

[solko-fbsd]

I already ported Zimbra to FreeBSD and I am happily using it.

BUT.

Integrating Zimbra into ports system is something ENTIRELY different. There are tons of files I need to write from scratch and one bad decision can cost you a day of retesting. Not counting the fact that I had to learn how ports system work (using and creating isn't the same sadly).
Furthermore, ports got some limitations and I won't bug developers with every single thing I encounter. It kinda wasn't designed in a way that part of installation goes to /opt/zimbra and other part to $LOCALPREFIX (/usr/local) (and I decided not to go that way anyway as it causes a lot of trouble in the end). It is wise to use /opt/zimbra as it will help with further upgrades as Zimbra team don't plan to change the prefix.
However, I try to install all might-be-shared-libraries into $LOCALPREFIX so that we can use maximum stuff from ports instead of Zimbra bundled and outdated ones.

If someone needs some help with testing.. just describe the symptoms. Usually, there is problem with java or a problem with postfix-smtpd dlopening the libplain/liblogin libraries (which I still haven't fixed in the binary package).

Testing in jail is perfectly possible if you can solve those stupid mysql/auth problems (jail specific behaviour).

As for ZFS, I am using ZFS on all my new servers but there is this known locking issue (restart only solution) which might not be suitable for big installations.

[iceburn]

Following your README

textbox1# pkg_add pkg/bash-4.0.10_2.tbz
pkg_add: could not find package libiconv-1.11_1 !
pkg_add: could not find package gettext-0.17_1 !

/usr/ports/shells/bash

so I compiled one from ports, its being installed in a jail (ezjail), so had to create the link in the basejail.

# pkg_add pkg/rpm-5.0.3_2.tbz
pkg_add: could not find package expat-2.0.1 !
pkg_add: could not find package beecrypt-4.1.2_1 !
pkg_add: could not find package pkg-config-0.23_1 !
pkg_add: could not find package lua-5.1.4 !
pkg_add: could not find package pcre-7.8 !
pkg_add: could not find package sqlite3-3.6.11 !
pkg_add: could not find package db45-4.5.20.0 !
pkg_add: could not find package libxml2-2.7.3 !
pkg_add: could not find package neon28-0.28.3 !
pkg_add: could not find package popt-1.7_5 !
pkg_add: could not find package xar-1.5.2 !

/usr/ports/archivers/rpm5

So installed from ports with WITHOUT_X11=yes and NOPORTDOCS=yes to cut down on the libraries required (esp. QT)

# pkg_add pkg/fetchmail-6.3.9.tbz
pkg_add: could not find package ca_root_nss-3.11.9_2 !

/usr/ports/mail/fetchmail


The ./install.sh works

But I'm having issues with my AMD64 server, when it starts importing the MySQL tables which causes it to lockup. So i haven't tested it any further. Seems like there is a problem with my gjournal or something which causes it to lockup.

Hopefully i figure out whats wrong or even remove the gjournal and go back to UFS+SU to test again.

[solko-fbsd]

Following your README

textbox1# pkg_add pkg/bash-4.0.10_2.tbz
pkg_add: could not find package libiconv-1.11_1 !
pkg_add: could not find package gettext-0.17_1 !

/usr/ports/shells/bash

so I compiled one from ports, its being installed in a jail (ezjail), so had to create the link in the basejail.

# pkg_add pkg/rpm-5.0.3_2.tbz
pkg_add: could not find package expat-2.0.1 !
pkg_add: could not find package beecrypt-4.1.2_1 !
pkg_add: could not find package pkg-config-0.23_1 !
pkg_add: could not find package lua-5.1.4 !
pkg_add: could not find package pcre-7.8 !
pkg_add: could not find package sqlite3-3.6.11 !
pkg_add: could not find package db45-4.5.20.0 !
pkg_add: could not find package libxml2-2.7.3 !
pkg_add: could not find package neon28-0.28.3 !
pkg_add: could not find package popt-1.7_5 !
pkg_add: could not find package xar-1.5.2 !

/usr/ports/archivers/rpm5

So installed from ports with WITHOUT_X11=yes and NOPORTDOCS=yes to cut down on the libraries required (esp. QT)

# pkg_add pkg/fetchmail-6.3.9.tbz
pkg_add: could not find package ca_root_nss-3.11.9_2 !

/usr/ports/mail/fetchmail


The ./install.sh works

But I'm having issues with my AMD64 server, when it starts importing the MySQL tables which causes it to lockup. So i haven't tested it any further. Seems like there is a problem with my gjournal or something which causes it to lockup.

Hopefully i figure out whats wrong or even remove the gjournal and go back to UFS+SU to test again.

From the README.FIRST:
I took the liberty of precompiling all the needed packages (-O2), however you will have to fetch the required dependencies all by yourself.
-snip-

However, I am wrong in the README.FIRST, one needs to be in the pkg directory for pkg_add to pick up dependencies properly :(.
I changed the README.FIRST and added a little troubleshooting FAQ.
http://mango.euba.sk/zimbra/zcs-install-FreeBSD_amd64/README.FIRST

As for mysql, I think it might be related to mysql+jail+auth problem. You will have to alter the mysql access list for this to work properly.
Please, check the logs whether mysql really hangs or whether mailboxd just keeps throwing exceptions 'cos of mysql.

[soylentgreen]

I'm currently downloading the source package to try the i386 version. Our production mail server is amd64, but the only machines I have to test with right now are 32 bit.

If Zimbra works as advertised and I can get away from The Horde, then I can get all those (whiny) iPhone and Blackberry users to stop using Google Calendar instead of the house system.

[iceburn]

Sat May 30 12:17:25 2009 *** Running as zimbra user: /opt/zimbra/bin/zmzimletctl -l deploy zimlets/com_zimbra_bulkprovision.zip
[] INFO: Deploying Zimlet com_zimbra_bulkprovision in LDAP.
[] INFO: Installing Zimlet com_zimbra_bulkprovision on this host.


I just sits there on the installation. I left it for about 12 hours and nothing.

[solko-fbsd]

Sat May 30 12:17:25 2009 *** Running as zimbra user: /opt/zimbra/bin/zmzimletctl -l deploy zimlets/com_zimbra_bulkprovision.zip
[] INFO: Deploying Zimlet com_zimbra_bulkprovision in LDAP.
[] INFO: Installing Zimlet com_zimbra_bulkprovision on this host.


I just sits there on the installation. I left it for about 12 hours and nothing.
Check the setup log file for:
*** Running as zimbra user: /opt/zimbra/bin/zmcontrol start

It should be located ~64 lines before those lines you pasted. Check if all services started normally. Check for running processes etc.
In any case try to rerun the setup.

I have never run into such problems myself :'(.

[evanc72]

Hey guys - checking in on this thread as I would love to see Zimbra on Freebsd. I only have i386 :-( so I guess I am stuck with linux at the moment.

solko-fbsd are you still working on this at all?

[solko-fbsd]

Oh well, kinda different problems nowadays (to-be-father-soon).
And then there is ZCS 6.0 which made my efforts kinda obsolete.

Please, poke me on ICQ 171813600, we should manage somehow.

I am going to look into it, I promise.

[evanc72]

Hey Congrats!!

Being a Dad comes first!!! I wish the best for you both, and a safe and happy birth!

As for zcs - it can wait. More important things to do for you now!

Cheers

[solko-fbsd]

I took a week off to make this happen:
http://wiki.zimbra.com/index.php?title=Building_GNR_on_FreeBSD_7

It's late, don't mind the mistakes (if any) :P.

Whole tree should be i386 ready but I need to test that first.

[solko-fbsd]

Whole tree should be i386 ready but I need to test that first.

Yup, running on i386 as well, uploading packages right now.
Check http://wiki.zimbra.com/index.php?title=Building_GNR_on_FreeBSD_7 for more details.

And please, start giving feedback finally! :P