Malleable Musings

January 4, 2009

More on email (unfinished post)

Filed under: Emails — Brendan @ 1:10 am

Just reviewing some stats on a couple of emails that were sent out just before the Christmas break. Interestingly I sent to the same email list a week apart, but from different domains. Email A came from our main london.ac.uk domain whilst email B came from a domain we set up specifically for emailing purposes (i.e. SenderID and DomainKeys have been set up alongside SPF).

Email A was sent at 4pm UK time on the 10th Dec whilst email B was sent exactly a week later. Long enough to make a difference I know (no two emails are ever the same) but the closest time period I’ve ever had for emailing the same list, so I thought it would be worth a comparison.

The basic stats were:

Email A %’ges Email B %’ges
All recipients 15,495 15,459
Opens 5581 36.018% 4,455 28.818%
Not Opened 9,348 60.329% 10,453 67.618%
Bounces 566 3.653% 551 3.564%
Clicks 1740 11.229% 156 1.009%
Unsubscribes 3 0.019% 7 0.045%

Now OK the content isn’t comparable and consequently I’m not going to try and compare click or indeed open rates because of this, Instead I wanted to look at the bounce rate.

My ESP allocates bounces into various types of soft and hard bounces. A hard bounce where the message is considered permanently undeliverable, but the recipient’s server has not provided a specific bounce reason. Hard bounces are automatically removed from your subscriber lists – so hard bounces. There were 34 hardbounces from email A which were removed from the mailing list for email B.

In email B there were 14 hard bounces (two of these email addresses had soft bounced in email A). In terms of domain distribution of these 14 hard bounces, 10 were free email accounts, e.g. Hotmail / Yahoo etc which presumably had expired in the week between the two sends.

I’m not too worried about the other four hard bounces as there were opens or clicks of email B from three out of the four domains.

The soft bounces are then categorised as follows:

Email A Email B
2 Auto Reply (AR) – These are soft bounces caused by an automatic response from the recipient, for example “Out Of Office” messages. The email is still actually delivered to the inbox, and once the subscriber opens the email (and is recorded), the bounce is removed from your reports.
169 161 General Bounce (GB) – The email server could not deliver your email message, but the bounce processing tool could not determine a specific reason for the bounce. Normally that is because the bounce message from the recipient’s server was very broad. We treat these as soft bounces. Example: “Subject: Undeliverable mail”
7 5 Mail Block – General (MB) – Indicates that the recipient’s email server is blocking email from our email server. Example: “550 Message REFUSED by peer”
10 11 Mail Block – Relay Denied (MBRD) – Indicates that the recipient’s email server is blocking email from our email server. Example: “551 relaying denied”
3 5 Mail Block – Spam Detected (MBSD)Indicates that the recipient’s email server is blocking your email because the message appears to have content that looks like spam. Example: “550 Possible spam detected”
186 190 Soft Bounce – Dns Failure (SBDF) – The email server is temporarily unable to deliver your message to the recipient email address because of a DNS problem. Example: “Host is unreachable”
82 82 Soft Bounce – General (SB)The email server is temporarily unable to deliver your message to the recipient email address. Example: “Connection timed out.”
71 73 Soft Bounce – Mailbox Full (SBMF)The email server is temporarily unable to deliver your message to the recipient email address because the recipient’s email box is full. Example: “Mailbox over quota”
4 8 Transient Bounce (TB) The email server temporarily can not deliver your message, but it is still trying. Example: “Warning: message still undelivered after 4 hours. We will keep trying until message is 2 days old”

I then ran some queries on soft bounces by domain and got the following data for the top ten domains that bounced:

pacific.net.sg 82 85 but one repeated – see below
singnet.com.sg 20 20
tstt.net.tt 15 15
aol.com 8 8
fish.co.uk 6 6
netvigator.com 6 2 seems to have improved dramatically
emirates.net.ae 5 9 but four repeated – see below
cec.eu.int 5 5
tm.net.my 4 2 seems to have improved dramatically
rediffmail.com 4 0 seems to have improved dramatically

In looking at these I then spotted that on Email B six email addresses were double counted (it looks like there was no double counting on Email A). One of the Autoreplies was also reported as SBMF and four TB’s (all on the emirates.net.ae domain) were also reported SBG whilst a further TB on a different domain was reported SBMF.

In total 18 email addresses that bounced on Email A did not bounce on Email B.

Whilst 11 email addresses that bounced on Email B (due to SB, GB and SBDF reasons) and a further 3 for spam detected did not bounce on Email A.

Having looked at various other metrics my conclusion is that Email B has probably performed marginally better for deliverability (but not by much). Howver from a design point of view Email B was always more likely to be caught by spam filters.

I’ve decided I need to investigate some of the worst performing domains and have decided to run a couple of A/B tests on the same email to see if it tells me more.

Advertisements

1 Comment »

  1. […] This is the first time for a while that I’ve run this test and I did so as I wanted to check that open and click through rates were still better when I didn’t use the the standard University domain to send email.  (I thought I’d better check following this unfinished post). […]

    Pingback by Why I don’t email from my work email account « Malleable Musings — January 7, 2009 @ 8:51 am


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: