Mailman 3

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 12,724
Messages: 39,314

Looks like it hasn't reached the ports yet. However, you can install 2.1.x: mail/mailman.
 
OP
B

balanga

Son of Beastie

Reaction score: 218
Messages: 4,033

I'm trying to log my progress for anyone interested in installing Mailman 3..

pkg install python3

pkg install py36-sqlite3

python3 -m ensurepip

pip3 install mailman

mailman info - this creates the mailman subdirectory tree in the current directory.

tbc...
 
OP
B

balanga

Son of Beastie

Reaction score: 218
Messages: 4,033

Trying to figure out the directory structure that Mailman 3 on FreeBSD should adopt. Here is what is used on Linux:-

http://docs.list.org/en/latest/config-core.html#configuring-filesystem-paths

I'd be interested in the FreeBSD Gurus view on how this should be configured on FreeBSD:-

Code:
bin_dir: /sbin
var_dir: /var/lib/mailman
queue_dir: /var/spool/mailman
log_dir: /var/log/mailman
lock_dir: /var/lock/mailman
etc_dir: /etc
ext_dir: /etc/mailman.d
pid_file: /var/run/mailman/master.pid
 

odhiambo

New Member

Reaction score: 2
Messages: 3

Hi,

I have followed your notes and installed MM3 on my FreeBSD 11.2.
I have opted to use /opt/mailman3 as the directory for MM3.
So I did:

cd /opt
mkdir mailman3
cd mailman3
mailman info

The directory has been populated with some data.

# Then from http://docs.mailman3.org/en/latest/prodsetup.html
pip3 install postorius
pip3 install hyperkitty
pip3 install mailman-hyperkitty
pip3 install uwsgi

All those installed successfully. However, I am not sure where they installed into :)

What next now?? Edit the var/etc/mailman.cfg??

Then how to launch the queue runner? Postorius? HyperKitty? etc?

TIA
 
OP
B

balanga

Son of Beastie

Reaction score: 218
Messages: 4,033

I gave up over a year ago, but following your post I will rekindle my interest :)
 

Relictrix

Member

Reaction score: 1
Messages: 66

Hi,
I tried mailman3 today. The story now is even shorter, when installing on 12.1-RELEASE-p10 after installing the pkg. It advises to execute service mailman oneinfo to set configuration etc. That already fails:

The port itself shows this:

To initialise Mailman's runtime directories, log files and database for
the first time, run service mailman info You can then start Mailman by
running: service mailman start

After doing so , I get this error:

Code:
root@vm1:/zroot/zdata/howdoyoudo/freebsd # service mailman oneinfo
Traceback (most recent call last):
  File
"[I]/usr/local/lib/python3.7/site-packages/pkg_resources/[/I]__init__.py", line
583, in _build_master
    ws.require(__requires__)
  File
"[I]/usr/local/lib/python3.7/site-packages/pkg_resources/[/I]__init__.py", line
900, in require
    needed = self.resolve(parse_requirements(requirements))
  File
"[I]/usr/local/lib/python3.7/site-packages/pkg_resources/[/I]__init__.py", line
791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (zope.interface 4.6.0
(/usr/local/lib/python3.7/site-packages),
Requirement.parse('zope.interface>=5.0'), {'mailman'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/mailman", line 6, in <module>
    from pkg_resources import load_entry_point
  File
"[I]/usr/local/lib/python3.7/site-packages/pkg_resources/[/I]__init__.py", line
3251, in <module>
    @_call_aside
  File
"[I]/usr/local/lib/python3.7/site-packages/pkg_resources/[/I]__init__.py", line
3235, in _call_aside
    f(*args, **kwargs)
  File
"[I]/usr/local/lib/python3.7/site-packages/pkg_resources/[/I]__init__.py", line
3264, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File
"[I]/usr/local/lib/python3.7/site-packages/pkg_resources/[/I]__init__.py", line
585, in _build_master
    return cls._build_from_requirements(__requires__)
  File
"[I]/usr/local/lib/python3.7/site-packages/pkg_resources/[/I]__init__.py", line
598, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File
"[I]/usr/local/lib/python3.7/site-packages/pkg_resources/[/I]__init__.py", line
786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'zope.interface>=5.0'
distribution was not found and is required by mailman

It's a fresh install and it fails on the first command, I wonder what I
should do next...
 

Relictrix

Member

Reaction score: 1
Messages: 66

Ah ok, nice to know next time i check the bug list before i launch something. Thanks for your reply Elazar!
 

dvl@

Aspiring Daemon
Developer

Reaction score: 104
Messages: 609

PR 225543 put Mailman 3 into the ports tree.

I have recently updated PR 250362 with a list of how to get Mailman running, but not necessarily functional.

My next step, porting Postorius to FreeBSD. If anyone is available to help, please let me know.
 

pboehmer

Active Member

Reaction score: 36
Messages: 150

We just moved/migrated our MM2 over to mail/sympa. We only had a handful of lists, so it wasn't too painful. List moderators like the interface better.
 

Oldno7

New Member

Reaction score: 4
Messages: 8

Hi everybody :
I've just finished the migration from mailman2 to a mailman3 for two servers and a bunch of lists (17 in one server 2 in the other). The servers are running 12.2-RELEASE (with python 3.7 -and 2.7- installed from ports).
I begun trying to install it from ports, following Mr. Langille recommendations but I faced a hell of dependencies issues. Then I followed the installation guide in https://docs.mailman3.org/en/latest/install/install.html, but out of docker or virtualenv environment. I made a mix of ports and pip installs as long I need.
The servers are jailed and both running apache24 but are proxied from a nginx server each one acting as firewall/proxy (that hosted the jails). For the wsgi process I installed mod-wsgi and uwsgi for hyperkitty
All is working like a charm now, but It was really difficult to find the correct config for all the processes involved.
If there is interest in this aproximation I will polish my notes and put them here.
 

Oldno7

New Member

Reaction score: 4
Messages: 8

Yes, please especially the missing dependencies and the configurations.
Well.. first of all I need to express my thankfulness to all of you who work in making FreeBSD the better and solid platform, and in this moment, with python 2.7 in its EOL, the migration of mailman2 to mailman3 was a big thing. Thanks a lot dvl@.
(and beg all of you your pardon if my english is misspelled or confusing: I think in spanish)

The start point:
2 servers upgraded to 12.2-RELEASE acting as firewalls/proxies in separated lines (one ADSL, the other optic fiber), with static IPs. Each of them had running sendmail + mailman2 + nginx and are the hosts of a number of jailed servers who serve Drupal or Wordpress sites (with apache24). The two had identical configurations. They have letsencrypt certificates (but some of them are proxied by cloudflare, too)
For each jailed server there is a mount point to mailman2 (/usr/local/mailman), the ports(/usr/ports) and the certificate from the host server(/usr/local/etc/letsencrypt).
All the mail treatment occurs in the main host with sendmail (by the way: I changed to sendmail from ports, as some time ago, two or three years ago if I recall it well, it was easier to config for auth mail and other things), but you can access to the web config in the jailed servers as they have mailman2 installed an accessible via the mount point. In the jailed servers the MTA was sendmail, but only with submit activated.
Lets say I have sol.mydomain.es (ADSL line) as the main host(local IP 10.0.1.1), and www.firstdomain.eu as jailed host (local IP 10.0.1.42) with 2 mailman lists. For the optic fiber line I have mercurio.mydomain.es (local IP 10.0.1.2) and www.seconddomain.es as its jailed host (local IP 10.0.1.51) with 17 mailman lists.

I hope it pictures well the panorama from I started...

As I said in my first comment, one of the jailed servers had just 2 lists in mailman, and it was the "target" to try and test the migration: they have not much email traffic. It served a Drupal site with apache24 (which means that I have to combine the new mailman3 Django project underlying).


[[[ I'm going to focus in this post in the tryings and "what didn't work well" things and in a separate post I will list the steps in the other server which work well at the end without all the frustration]]]

- The first thing I thought I had to do was to change the MTA in the jailed server (www.firstdomain.eu - 10.0.1.42): mostly all the help guides are prepared to use postfix as MTA, so I did a "portsnap fetch update" and installed postfix form ports. I followed https://www.freebsd.org/doc/handbook/mail-changingmta.html . It needed some adjusts for mailman3 but I applied them after (we'll come back to the postfix and sendmail in the main server config in the "all went well" post)

(WARNING: all the python installs from ports where with FLAVOR=py37 as I have a running python 2.7 apps yet)

- I followed the post of dvl@ in this thread, made a make install of /usr/ports/mail/mailman3 and
it went well. As he mentioned, I was expecting dependencies problems (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=250362). For the steps in the whole installation I tried to follow https://docs.mailman3.org/en/latest/install/virtualenv.html#virtualenv-install, but out the virtualenv enviroment (I've noticed some folks having serious problems with virtualenv in freebsd, and I have no experience with it).
After I had mailman3 running I tried to install and config the Django ports. I found that the perl version was too low (5.26), and then upgraded all perl dependencias to 5.30. And here I fell into hell: it needed upgrade rust (which I upgrade from pkg, because from ports it will take DAYS to compile). It had conflicts with ImageMagick and php-gd... and many many more in a loop.

This was the moment I decided to uninstall Django from ports and try to "pip install" what I need. At the end I reinstalled all the mailman apps from pip and configured all thereafter.
Next post I show the steps that went well from the beginning.
 
Last edited:

Oldno7

New Member

Reaction score: 4
Messages: 8

As I said in my previous post (please review the first part of it to see all the picture), after I made a succesfull mailman3 installation in one server with 2 mailman2 lists, I did it again with a jailed server with 17 mailman2 lists.
The servers: mercurio.mydomain.es (local IP 10.0.1.2) and www.seconddomain.es as its jailed host (local IP 10.0.1.51) with 17 mailman lists.
The scope:
postfix config files in /usr/local/etc/postix (I attach main.cf as postfix-main.conf with all comment lines stripped)
postfix virtual users in /usr/local/etc/postfix/virtusertable
mailman3 services in /usr/local/mailman3
/usr/local/bin/python3.7 has a ln -s to /usr/local/bin/python3 (I had python linked to the 2.7 version yet)


In the jailed server (www.seconddomain.es)
- portsnap fetch update

- install /usr/ports/mail/postfix

- followed the changing MTA from sendmail (only submit) to postfix guide https://www.freebsd.org/doc/handbook/mail-changingmta.html

- created user for sasl2 authentication in /usr/local/etc/postfix/sasl_passwd and postmap it
[mercurio.mydomain.es]:587 mailoutuser: passoutuser
(there is no space after the colon, but this wysiwyg translate as a :p, so I put one here)
(you have to create the user in the main server with "saslpasswd2 -c" )

- installed /usr/ports/devel/py-pip with FLAVOR=py37

- did a "ln -s /usr/local/bin/py37-pip to /usr/local/bin/pip" in order to clarity and follow guides

- pip install mailman

- installed /usr/ports/textproc/sassc and did an "ln -s /usr/local/bin/sassc /usr/bin/sassc" for django to find it (important)

- pip install mailmanclient

- created user mailman3 with the same gid and uid as mailman (1113:1113 in my system) and with /usr/local/mailman3 as home .

- mkdir /usr/local/mailman3 & chown mailman3:mailman /usr/local/mailman3

- install /usr/ports/devel/git

- cd /usr/local

- git clone https://gitlab.com/mailman/mailman-suite.git
That clones mailman-suite and bring us some files to get config

- cd mailman-suite/mailman-suite_project
- su -m mailman3 -c " cp manage.py, urls.py and wsgi.py /usr/local/mailman3"
(you can now delete /usr/local/mailman-suite as is no longer needed)

I attach the mailman.conf file that would be /usr/local/mailman3/var/etc/mailman.cfg

[[[Post edition 12/27/20:
I've found that I missed to define one important thing in mailman.cfg
The site owner as it is the "from" address which sends you the unsuscribe notifications, etc.
You can see all the config possibilities in https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/config/docs/config.html
I've added these lines in the top of my mailman.cfg file
###########
[mailman]
site_owner: webmaster@seconddomain.es
default_language: es
###########
(the second line, obviously, is not necessary for your config)
]]]

- installed /usr/ports/databases/py-pymysql as I want that mailman uses mysql database (and let Django with Sqlite3)

- Created a "mailman" database in mysql and granted permissions for IP, localhost and host "www.seconddomain.es" accessing with "mysql -p -u root" and executing (I just put here the last grant)
"GRANT all privileges on mailman.* to 'mailman3'@'www.seconddomain.es' IDENTIFIED BY 'databasepass' WITH GRANT OPTION;"

- installed
pip install postorius
pip install hyperkitty
pip install mailman-hyperkitty
pip install whoosh
"python3 -m django --version" shows 3.0.11 (that's it)

- installed /usr/ports/databases/py-sqlite3

I attached the settings.py.conf needed that would be /usr/local/mailman3/settings.py for Django
(Obtained and modified from the previous git run)

su -m mailman3 -c "mkdir /usr/local/mailman3/logs"
su -m mailman3 -c "touch /usr/local/mailman3/logs/mailmansuite.log"

Last thing:
As I installed postfix in /usr/local, mailman cannot find the postmap command to hash de virtual tables, then I did (using bash)
for fic in `ls /usr/local/sbin/post*`; do ln -s $fic /usr/sbin/`basename $fic`; done

There we go: (REMEMBER ALWAYS DO THINGS AS MAILMAN3 USER)
- cd /usr/local/mailman3

- su -m mailman3 -c "python3 manage.py migrate"
(This creates all the things needed)

- su -m mailman3 -c "python3 manage.py collectstatic"

- su -m mailman3 -c "python3 manage.py createsuperuser"
(the admin: in my example is myuser@mydomain.es)

**** at this point you have all the django settings made. Before doing the apache24 part I have to say a couple of things:
In the virtualenv install guide stated you can start an instance of server with a
"python3 manage.py runserver". That's good to test the settings... but can confuse you because it's not needed at all if you have a "real server" like apache or nginx serving http. At least it have me missleaded for many days.
The hyperkitty config base path: it took me much frustrating time of confusion as in the guides it stated just as "http://localhost/hyperkitty"... and it not archives any mail at all untill I understand that it have to be as it shows in my config file (well, I changed the access for "mailman3" but it would be preferable if it was noted as, say, "http://localhost/hyperkitty/hiperkitty"
****
For the apache24 part I decided to use the mod-wsgi (and uswgi binary).
- pip install mod-wsgi
(with "mod_wsgi-express module-config" it shows the conf needed.)

- pip install uwsgi

- Included these lines in the "loadmodule" part of /usr/local/etc/apache24/httpd.conf
#####
LoadModule wsgi_module "/usr/local/lib/python3.7/site-packages/mod_wsgi/server/mod_wsgi-py37.so"
WSGIPythonHome "/usr/local"
WSGIDaemonProcess hyperkitty threads=25 python-path=/usr/local/mailman3 user=mailman3 group=mailman
WSGIProcessGroup hyperkitty
#####
- Included this line in the correct places of my /usr/local/etc/apache24/extra/httpd-vhosts.conf (in the listen 80 and 443 servers)
Include "/usr/local/mailman3/apache24.uwsgi.conf"
(I attach my apache24.uwsgi.conf. Remember I have a Drupal site enabled here)

- service apache24 restart

- initiated the uwsgi with a
/usr/local/bin/uwsgi --ini /usr/local/mailman3/uwsgi.ini &
(beware the "&" backgrounding it)
As I can only attach 5 files I put it here
####### start of /usr/local/mailman3/uwsgi.ini
# uwsgi.ini
#
[uwsgi]
# Port on which uwsgi will be listening.
http-socket = 0.0.0.0:8000
# Move to the directory wher the django files are.
chdir = /usr/local/mailman3/
# Use the wsgi file provided with the django project.
wsgi-file = wsgi.py
# Setup default number of processes and threads per process.
master = true
process = 2
threads = 2
# Drop privielges and don't run as root.
uid = www
gid = www
# Setup the django_q related worker processes.
attach-daemon = ./manage.py qcluster
# Setup the request log.
req-logger = file://usr/local/mailman3/logs/uwsgi.log
# Log cron seperately.
logger = cron file://usr/local/mailman3/logs/uwsgi-cron.log
log-route = cron uwsgi-cron
# Log qcluster commands seperately.
logger = qcluster file://usr/local/mailman3/logs/uwsgi-qcluster.log
log-route = qcluster uwsgi-daemons
# Last log and it logs the rest of the stuff.
logger = file://usr/local/mailman3/logs/uwsgi-error.log
####### end of /usr/local/mailman3/uwsgi.ini

And thats all about mailman 3 configuration. You can start all with two commands
/usr/bin/su -m mailman3 -c "/usr/local/bin/mailman -C /usr/local/mailman3/var/etc/mailman.cfg start"
/usr/local/bin/uwsgi --ini /usr/local/mailman3/uwsgi.ini &
I created a mailman3.sh to start, stop and restart all when needed if someone wants it I'll attach it in other post.

Then you have to create the lists in postorious and migrate following the guide in https://docs.mailman3.org/en/latest/migration.html
It works very very well (I migrated all the lists without any problem)

One thing I did is declare as domain "seconddomain.es" and as alias "www.seconddomain.es" in the postorious config for adding domains.

You do remember that www.seconddomain.es was jailed, isn't it? And the MX server in DNSs for seconddomain.es is mercurio.mydomain.es. That implies that we need to "resend" the mail for the lists from mercurio.mydomain.es to www.seconddomain.es (as with mailman2 was mercurio the server that manages mailman2 lists).

To do that I use a virtual user table in postfix I named "virtusertable" (like in sendmail). That's why I append mail.seconddomain.es in muy destinations in postfix.

Say you have the list one_list@ seconddomain.es

I created in /usr/local/etc/postfix/virtusertable the following lines:

# Virtual mappings for the @seconddomain.es domain.
one_list@seconddomain.es one_list@www.seconddomain.es
one_list-bounces@seconddomain.es one_list-bounces@www.seconddomain.es
....
one_list@mail.seconddomain.es one_list@www.seconddomain.es
one_list-bounces@mail.seconddomain.es one_list-bounces@www.seconddomain.es
....
I did not copied here all the lineas needed, but you can get them easily doing a
"cat /usr/local/mailman3/var/data/postfix_vmap >> /usr/local/etc/postfix/virtusertable"
and adding the "mail.seconddomain.es" entries for each list.

And don't forget to postmap /usr/local/etc/postfix/virtusertable (and service postfix reload after that)

That sends all email for lists to the correct addresses

To manage that in mercurio.mydomain.es I configured the local-unbound process, and, for sendmail, configured /etc/mail/mailertable and /etc/mail/virtusertable.

in mercurio:
/var/unbound/unbound.conf:
# www.seconddomain.es local zone
private-domain: "www.seconddomain.es"
local-zone: "www.seconddomain.es." static
local-data: "www.seconddomain.es. IN A 10.0.1.51"
local-data: "mail.seconddomain.es. IN A 10.0.1.51"
local-data: "www.seconddomain.es. IN MX 10 mail.seconddomain.es."
local-data-ptr: "10.0.1.51 www.seconddomain.es."


/etc/mail/mailertable:
www.seconddomain.es smtp:[mail.seconddomain.es]

/etc/mail/virtusertable:
one_list@seconddomain.es one_list@mail.seconddomain.es
one_list-bounces@seconddomain.es one_list-bounces@mail.seconddomain.es
... (rest of the hooks for the list)

The reason to do it this way (using a "fake" mail.seconddomain.es server) is because I didn't achieve to resend emails in other way as mailertable did not work.

And that's all...
 

Attachments

  • potsfix-main.conf
    2.3 KB · Views: 55
  • mailman.conf
    1.9 KB · Views: 74
  • mailman-hyperkitty.conf
    173 bytes · Views: 54
  • settings.py.conf
    8.3 KB · Views: 57
  • apache24.uwsgi.conf
    1.4 KB · Views: 64
Last edited:

Oldno7

New Member

Reaction score: 4
Messages: 8

A few days ago I did an upgrade of the mailman3 related apps (mailman from 3.3.2 to 3.3..3, e.g.). I write down here the process to fulfill a successfull upgrade (for anyone who access in the future this posts and have the same environment):
I did, as superuser:
pip install --upgrade django-mailman3 mailman mailman-hyperkitty mailmanclient postorius django hyperkitty

Then (be aware that mailman3 here is the user that runs mailman), in the mailman3 main directory did:
su -m mailman3 -c "python3 manage.py collectstatic"
su -m mailman3 -c "python3 manage.py migrate"

There are recommendations that you execute also a
su -m mailman3 -c "python3 manage.py compress"
su -m mailman3 -c "python3 manage.py compilemessages"

But I didn't and all my mailman3 lists are working as before. I think it depends of your config.

[[[Post edition on 02/16/21: If you need to update your translation as e.g. using spanish you may want do the "compilemessages" run. But if you, as myself, ended installing all using pip, you will find an error if you try to do as I put it above. For the sake of python doing it well, you have to move to each python package directory and do it using django-admin command as superuser.
E.G. you have python3.7 installed. it had to be done, at least for postorius, hyperkitty and django_mailman3:
cd /usr/local/lib/python3.7/site-packages/postorius
django-admin compilemessages
...
and so on for the other two.]]]


P.S.:
I couldn't find any guide about this process of upgrading and I just did the "migrate" part the first try. It showed that you cannot access the lists via postorious falling in a extrange exception. After reinstallating all, I realized that the "collectstatic" part was necesary because it's not invoked in the "migrate" run.
 
Last edited:

Relictrix

Member

Reaction score: 1
Messages: 66

Hi,
I will have to catch up with this thread later, I am still planning to do this.
Best Regards,
R
 

odhiambo

New Member

Reaction score: 2
Messages: 3

Oldno7 where is the startup script? The one for /usr/local/etc/rc.d/. And where is the cron file?
I am about to embark on this.
I had hope the FreeBSD port would have encompassed all this by now.
 

dvl@

Aspiring Daemon
Developer

Reaction score: 104
Messages: 609

I gave up on moving to Mailman 3, mostly because of the complexity. My work there has ended.

I will be moving to mlmmj, much like the FreeBSD mailing lists are moving there.
 

hruodr

Aspiring Daemon

Reaction score: 283
Messages: 896

I gave up on moving to Mailman 3, mostly because of the complexity. My work there has ended.

I will be moving to mlmmj, much like the FreeBSD mailing lists are moving there.
I did not understand why the other needed to be so complicated and installed mlmmj from the beginning.

Why the hell needs mailman for example to run a daemon?!
 
Top