Mailman

Components

Mailman3 has multiple components. Each doing a different thing. This is a simplified overview. For a more visual diagram, see the upstream architecture overview.

mailman3 is the service that handles emails and talks to MTA (in our case exim4) through LMTP/SMTP
mailman3-web or postorius is a django-based web service that handles settings, subscriptions, etc. It doesn't do anything on its own (except user management) but it talks internally to the mailman3 service through REST API.
hyperkitty (being used in the same webservice) is responsible for archives. It handles search, etc.
Search backend: Is xapian and the search index is in /var/lib/mailman3/web/fulltext_xapian_index
Databases: Two databases both in m5:
- mailman3 which mailman's core service talks to. It holds memberships, lists, etc.
- mailman3web which postorius and hyperkitty talk to. It holds archive of all emails and is rather big.
- MTA: It's exim4 (see Mailman/MTA for more information)

Who has access?

Mailman3

The following people have "superuser" access via the web UI for admin access to all lists:

Legoktm
Ladsgroup
Quiddity (Nick Wilson) (WMF)
"Administrator" (all SRE members or users with root)
"WMFOffice" (Trust and Safety team)

How To

Create a mailing list

Go to https://lists.wikimedia.org/postorius/lists/new/ (as an administrator)
or ssh to lists1001.wikimedia.org and run "sudo mailman-wrapper create --owner OWNER [email protected]"

Consider if you need to notify Trust and Safety when creating new mailing lists.

The only part of new mailing lists that can't be easily changed is its name. Pleas make sure it follows naming standards before creating.

Disable or re-enable a mailing list

Remove owners and add [email protected] as owner
- This is important, otherwise we can't find it and change/fix its settings.
Set all emails to be discarded (both member and non-member)
Ban ^.*@.*
Add to description that this mailing list is disabled like [Disabled] or [Archived]

Remove a mailing list

Note: unless you know what you're doing, don't actually do this. It most likely does not do what you want. Just disable the list (see above) instead.

Go to https://lists.wikimedia.org/postorius/lists/LISTNAME.lists.wikimedia.org/delete (as an administrator)
or ssh to lists1001.wikimedia.org and run "sudo mailman-wrapper remove [email protected]"

Remove a message from the mailing list archives

Delete the whole thread

Go to the thread in hyperkitty (example).
Click on "Delete this thread"

Delete a single message

Go to the message in hyperkitty (this can be tricky, go to the thread, click on the chain icon aka "Permanant link to this email"). example
Click on "Delete this message"

Export a listing of all subscribers to a mailing list

Go to members settings: https://lists.wikimedia.org/postorius/lists/LISTNAME.lists.wikimedia.org/members/member/
Click on "CSV export"

Check if an email address is subscribed to any lists

Remove an individual from all mailing lists

ssh to lists1001.wikimedia.org
Run "sudo mailman-wrapper delmembers -m EMAIL_ADDRESS --fromall"

Reset a user's password

ssh to lists1001.wikimedia.org.
Run "sudo mailman-web changepassword USERNAME"

Rename a mailing list

Import from Google Groups

It is straightforward to migrate a list from Google Groups to Mailman. A list owner in Google should use Google Takeout (see documentation) to export the member list (members.csv) and archives (topics.mbox). File a task requesting a new list and include a link to the Takeout result (if the list contents are private, provide this information to the Mailman admins/SRE team privately).

Create the new list in Mailman like normal, then use the "Mass subscription" form at <https://lists.wikimedia.org/postorius/lists/LISTNAME.lists.wikimedia.org/mass_subscribe/> to import the existing subscribers.

Copy the mbox to the mailing list server and run:

sudo mailman-web hyperkitty_import -l [email protected] /path/to/topics.mbox

This might take a while because of the "Synchronizing properties with Mailman" step, that's fine.

Then update the search index:

sudo mailman-web update_index_one_list [email protected]

All done! Make sure to have the list owner in Google disable the group there, and remind subscribers to update the lists' email address to LISTNAME@lists.wikimedia.org.

Add a list to Gmane

A new list can be added to Gmane very easily if the first message has not already been sent or history is not important. If the history needs to be imported, a ticket should be file in Phabricator with projects 'Wikimedia-Mailing-list' and 'operations' (easy link stating which list needs to imported and to which Gmane group if it exists already.

Docs and links

Site Administrator docs on gnu.org

Authorized recipients for [email protected]

Anyone with a signed NDA can be on the [email protected] mailing list. For a list of signed NDAs for staff/contractors there is T83783, WMF-NDA and ldap/nda (or ldap/wmf).

Migrate servers

See Mailman/Migration for comments made about past migrations (lily to sodium, sodium to fermium, fermium to lists1001).

Upgrading Mailman3

When upgrading Mailman3 packages for a version that includes schema changes, use the following process.

Downtime host in icinga
Disable puppet
Manually stop mailman3 and mailman3-web systemd services
Run apt to install the new packages
Run sudo mailman-wrapper help to apply the mailman3 schema updates (yes, running just the "help" command will apply the schema changes)
Run sudo mailman-wrapper help again to make sure the schema update didn't obviously break anything.
Run sudo mailman-web migrate to apply the mailman3 schema updates
Clear static CSS/JS caches: sudo mailman-web collectstatic --clear --noinput && sudo mailman-web compress
Start mailman3 and mailman3-web systemd services again, keep an eye on errors in the logs
Re-enable puppet and remove icinga downtime if it hasn't expired yet.

Post upgrade:

Verify that the messages the MediaWiki Installer screen-scrapes haven't changed.

Block abusive traffic

Use profile::lists::web_deny_conditions in private puppet to block at the Apache level (see gerrit:736211)
Use the standard abuse_networks in private puppet which will block at the ferm/network level (see gerrit:736552)

Configuration details

The new Mailman3 setup lives on lists1001. Some state is under /var/lib/mailman3/, the global configuration is in /etc/mailman/ via Puppet.

The mail server used is Exim, the web server used is Apache.

Packages

We are currently using custom backported packages of the mailman3 suite. A few patches have been cherry-picked from upstream to fix issues we've run into. The packaging Git repositories can be found on apt1001 in legoktm's home directory.

There is currently one change that is hotpatched and not in the package yet.

Mailman setup

TODO: Add

Mail server setup

See Mailman/MTA

Web server setup

To get Mailman running with Apache the puppet class role::lists is applied which uses module mailman. Puppet module mailman uses puppet module Apache which sets up Apache and the site config is in the template lists.wikimedia.org.erb.

Backups

lists1001 is backed up to backup1001 using Bacula. All of /var/lib/mailman3 is backed up, which potentially includes on-disk templates and the search index. We do not back up /var/lib/mailman3/web which comes from the package and digests that haven't been sent yet.

The database lives on m5, which is also regularly backed up into Bacula, too.

Tested failure modes

Because mail delivery and transport should be reliable, I have tested what happens in certain failure modes, e.g. when SpamAssassin's spamd daemon is not running.

Spamd not running

Because of the /defer_ok modifiers in the Exim ACLs, Exim will act as if no spam filtering attempts are made when spamd is not running, and will accept the message. The following lines are logged:

spam acl condition: warning - spamd connection to 127.0.0.1, port 783 failed: Connection refused
spam acl condition: all spamd servers failed
H=xxx.xxxxxxx.xx [xx.xx.xx.xx]:xxxx I=[145.97.39.157]:25 U=exim Warning: ACL "warn" statement skipped: condition test deferred

Mailman not running

If the Mailman queue runner daemons are not running, incoming messages will still get delivered to the Mailman queue by Exim. However, nothing else will happen until the Mailman processes are started.

Spam fighting

See Mailman/Spam fighting

Templates

Most emails sent out by Mailman3 can be customized using "templates". You can see the default English templates. You can edit them by visiting https://lists.wikimedia.org/postorius/lists/YOURLISTNAME.lists.wikimedia.org/templates. These are translatable on translatewiki.net.

It may also be useful to read the template documentation (the code parts can be ignored).

Monitoring

See Mailman/Monitoring

Mailman2 data

Most private Mailman2 data is backed up in bacula in a long-term backup (details). See how to restore a file.

See also