Composr Tutorial: Understanding and configuring e-mail

Written by Chris Graham (ocProducts)
E-mail can be a complex thing. There are a number of e-mail protocols with standards that seem cobbled together, and there are a wide array of different tools that work with e-mail. Making everything compatible can be quite a challenge.


E-mails in Composr

Composr constructs its e-mails using language strings: each different e-mail is built from a different language string. These strings are written in Comcode. Composr sends out e-mails in dual format – both HTML and plain text, so that people can disable HTML in their e-mail software if they wish to. Plain text versions are made by automatic tidying-up of the Comcode (i.e. making it a little more human-readable), and HTML versions are made by parsing the Comcode to HTML and then putting that HTML inside the MAIL.tpl template. Composr goes to great lengths to reduce the chance of e-mails being marked as spam, and embeds all CSS and images inside the e-mail instead of linking (so that the user does not need images enabled to see them – most users do not as it can aid spammer tracking).

Just taking one example, the 'MAIL_NEW_PASSWORD' language string from the cns language file, which is:

Code

The password for your account (username: {4}) has successfully been changed to '{1}'. You may log back into {3} from...

[url="{2}"]{2}[/url]


If you wish to change your password to something more memorable you can do so by [url="{5}"]editing your account[/url].

You can see it is fed with various parameters, and is written in Comcode.

Configuration options

Pertinent configuration options are:
  • "Link to images in e-mails rather than embed" – by default this option is disabled, and when disabled makes Composr embed all images directly within e-mails rather than remotely linking to them. This means that users will see images immediately without having to grant permission for their e-mail client to download them.
  • "Use true 'from' address" – by default this option is disabled, because Composr sends e-mails from a separate address to the reply-to address, to reduce chance of e-mails being considered fradulent.

Mail server overview

Image

(Click to enlarge)

First, I will start with a brief overview of how e-mail works. This section may be far more than you ever need to know, so don't worry if you don't follow it all.

Consider that an e-mail address is composed of two parts: an account name, and a domain name (with an '@' symbol to separate them).

This is a simple thing to understand but let's look at some more detail. The first question is 'where does it get delivered to?', and the answer is 'the server identified by the MX record of the domain name that it is being sent to'. To deliver an e-mail to someone@example.com we would look up the MX record (a type of DNS record) for the example.com domain, and thus find the IP address of the destination server. This actual delivery process is performed by the 'SMTP' server, otherwise known as an 'outgoing e-mail server'. When you send an e-mail from a mail client (be that a desktop client, a webmail client, or a webapp like Composr), it is sent to the outgoing SMTP server to be dispatched. That server will put the message in a queue, and then it will (in the SMTP server's own time) send it on to SMTP server on the IP address of the MX record for the domain name ('destination e-mail server'). If it cannot be delivered it is kept in the queue while a few re-tries are attempted over a few days. The destination server will then then deliver the e-mail to the account specified in the e-mail address, and give a bounce e-mail if no such account exists (assuming it hasn't been set up to forward the e-mail to another account or address).

Relaying (advanced)

The procedure we described above is called 'relaying' because it is a two-step process: there are both outgoing and destination e-mail servers involved. Usually relaying is only permitted for e-mail senders who are trusted by the outgoing e-mail server, so that the outgoing e-mail server can't be used for purposes of sending spam e-mails. A user can only send through an e-mail server that they are allowed to relay through (and a common work-around to this is setting up one's own SMTP server, which can run on your own computer, or by writing special software that sends directly to the destination SMTP server without requiring relaying).
Sometimes SMTP servers relay over more than two steps. For example, it is possible to configure an e-mail server that relays all the e-mail that does not belong to local domains to another e-mail server. Of course, the server relayed to would have to be configured to allow this.


What I have just described is the primary mechanism for e-mail. However, there is a secondary mechanism – actually being able to read e-mails from an inbox (SMTP will populate an inbox but provides no way to actually read it). This are three common ways to read inboxes:
  1. Using the IMAP protocol (which is designed to permanently store e-mail on the server)
  2. Using the POP3 protocol (which is designed to transfer e-mail from the server to the user's e-mail client)
  3. Accessing the mail box directly (webmail often does this) as do UNIX command-line utilities that run directly on the server

It is important to understand that IMAP/POP3/webmail are entirely separate from SMTP itself, except for two areas:
  1. They access the same mailbox that SMTP writes to
  2. SMTP often whitelists the IP addresses of users who have recently logged into POP3 or IMAP to say that relaying should be allowed from those IP addresses (this is one mechanism for relaying to be allowed, another is authenticated SMTP, and another is from-address whitelisting)

SMTP configuration in Composr

There are two separate issues for us to consider when it comes to Composr:
  1. Whether we will want (i) Composr's SMTP-connection code to run, or (ii) PHP's SMTP-connection code.
  2. Which SMTP server PHP or Composr is connecting to. Neither Composr nor PHP include an actual SMTP server, so you're always going to be configuring one of them to connect to an actual SMTP server. The issue is whether that is your server's own SMTP server (assuming you have one) or whether it is another one (usually your hosting provider's). If you're on a Linux/UNIX server you have no choice but to use your server's own SMTP server.

It is usually best to rely on PHP's SMTP-connection code, so it can be managed on a server level. However there are two situations where this is not workable:
  1. PHP doesn't support SMTP authentication, so if the only e-mail server available requires this, you'll need to use Composr's SMTP-connection code (which does).
  2. If the PHP mail configuration is misconfigured or faulty and you can't repair it (see below).

Composr's SMTP-connection code is configured from the Configuration module (Admin Zone > Setup > Configuration > Site options). If the SMTP server hostname is left blank (the default), Composr relies on PHP's SMTP-connection code.

Avoid getting spam-blocked

When a website sends out e-mail there is always a risk that it could get blocked by spam filters. Whether this happens largely depends on the configuration of spam filters at ISPs and on user's own computers, but there also some general causes.

Specific issues can be:
  • If your server is on a spamlist. A good tool to check for this is: http://www.mxtoolbox.com/blacklists.aspx. Also check to see if any bounce messages come back that talk about your server being blocked as a spammer.
  • If your "Website e-mail address" is for an e-mail address hosted on another server and an SPF record exists for the domain does not grant your web server permission to use the address for sending out mail. Common e-mail services like gmail often have this problem. If this might be the case, you either need to get the SPF record amended to cover your server (impossible for a common service), or use a different "Website e-mail address". Note that Composr uses the "Website e-mail address" as the "From" address in all outgoing e-mails, but the reply addresses depend on context (often they are the "Staff address", but they could also be the address of the member who caused the e-mail to be sent.
  • Ensure you have reverse DNS available on your server's IP address. This is the address outbound SMTP connections are made from.
  • Ensure your server is giving a valid HELO DNS address when it makes outbound SMTP connections, not something generic like localhost. Preferably it will use your actual domain name, or a subdomain there-of, but this is not mandatory.
  • Ensure the PHP mail.add_x_header option is set to off (it flags up on SpamAssassin).
  • If your messages look like spam, which sometimes can happen inadvertently. Very carefully check your spam folder. Spam filters typically run a complex set of calculations to detect if something is 'spam'. It could well by a domain SPF setting is wrong, and combined with Composr e-mails being more complex than some other software, that knocking it over a spam threshold. That is just one of many possibilities that should be looked into if it is indeed a spam-filtering issue. A good tool to check spam filtering scores is: http://www.softexsw.co…/online-spam-score-check/
  • It's unlikely, but it could be some host/software-specific bug. You can open a bug report if you're willing to give the developers access to run tests on your server, after you've checked it's not a spam folder, and only if you're not on a low-quality free webhost.
  • If you've customised the MAIL.tpl template ensure you've also customised the MAIL.txt template to have the same text (it flags up on SpamAssassin if the words are inconsistent).
  • Make sure the Return-Path in e-mails is correct and a valid account, so that bounce/receipt emails don't themselves bounce. You may need to enable the "Pass website e-mail address to 'sendmail'" option.

Positive advice:
  • Generally it is advisable to set up SPF, as it provides a positive signal that your server is not a spammer. Set both SPF and TXT record types for maximum compatibility (set them to the same values).
  • One very effective way to stop your messages being marked as spam is to persuade your visitors to add your staff address to their contacts list. Spam checkers usually will not block mail sent from someone on their contacts list. If their e-mail provider has a "Safe senders" list, that's even better – Microsoft's e-mail services have this. Microsoft's e-mail services do over-block unknown e-mail servers or if users aren't reading your e-mails for long. If you're blocked, your message may not even go into the user's spam folder.
  • Get DKIM configured on your server.
  • If you're really stuck with your e-mail server being blocked, you could use a service like Mandrill. Mandrill may require the "Enable Blind CC (BCC)" option to be turned off, as we have had a report of it not working on Mandrill, but that they provide an account setting to make CC behave like BCC.

Queues and debugging

Composr has config options for:
  1. turning on a mail queue (for efficient delivery).
  2. keeping stuff in the queue until you let it out (when testing a site, the "E-mail debug mode" option).

The queue can be viewed from:
Admin Zone > Audit > E-mail queue/log

All (†) e-mails are logged for 14 days in the same system, regardless of whether the queue is on or not. This is very useful for analysing what your site has been sending out.

† Except support ticket e-mails if IMAP integration is enabled, newsletter e-mails, password reset e-mails, and some other very high security or high volume e-mails.

Concepts

SMTP
The e-mail protocol for sending e-mail and delivery between servers
IMAP
A protocol for accessing e-mail stored on a server
POP3
A protocol for downloading e-mail from a server
SPF
A special domain name record that indicates which servers are authorised to send e-mail for the domain name the record is for
MX
A type of DNS record for identifying mail servers for a domain name

See also


Feedback

Please rate this tutorial:

Have a suggestion? Report an issue on the tracker.

Back to Top