• by George Bilbrey , Op-Ed Contributor, March 14, 2012

A question popped up recently on a private email list I participate in about the validity of seed lists to determine inbox placement rates. To answer that question, it helps to take a look at the evolution of the practice of deliverability and also give a peek into what the future might hold.

In the earliest days of email, monitoring your deliverability was relatively simple.  Most ISPs had no spam folder, so if a message got rejected, you’d know it from the high amount of bounces in your bounce logs, or because your IT person was calling to say that the server was down because of the flood of rejected addresses returning back to your mail server through bounce messages.  And if you weren’t monitoring your bounce logs or bounce mailbox, you’d surely notice when your response rates were below average.  Then as spam increased, ISPs reacted by creating the spam folder and feedback mechanisms for subscribers like the “this is spam” button.  If you garnered enough spam votes or your message looked like spam based on content, the ISP would deliver the message to the spam folder. 

Marketers now had a new problem. They had no idea if their emails wound up in the spam folder or inbox, since both were reflected in a delivery report as “accepted for delivery.”  Additionally, most ISPs started disabling images by default so it became more difficult to rely on the open rate as a proxy for deliverability. And a low open rate might be indicative of other problems such as a poor choice of subject lines, a bad offer, an old list – too many reasons to even count here.

In answer to this new challenge, companies created email accounts known as seed lists at all the popular ISPs -- a tool used to report if an email campaign was delivered to the inbox or spam folder.  Marketers continue to rely on seed lists determine if they have a problem with their commercial mails being delivered to the spam folder. Seeds are a good proxy for overall delivery because most of the filtering is done at a “global” level.  The rules used at the ISP are applied to every message equally, regardless of which subscriber is receiving the message.  If seed addresses didn’t receive the message (or showed they went in the bulk mail folder), the overall campaign likely has a problem.

From the ISP perspective, global reputation-based filters have worked really well. Most ISPs only deliver a negligible amount of spam to the inbox.  Generally speaking, most of the emails delivered to the inbox are things people signed up for. But for some people, even this legitimate mail is it too much to handle.  To help their users manage the sometimes-crushing volume of legitimate messages, ISPs have been moving toward engagement-based filtering.  Hotmail has gone on the record saying that how an individual subscriber interacts with an email can influence where it’s delivered.  Gmail also rolled out Priority Inbox, which classifies emails as important based on whether it’s someone users know, or how they’ve engaged with it, such as replying to an email.  This kind of filtering leads to a natural question: does seed list deliverability monitoring still work?

Yes and no.

Our seed lists cover 142 ISPs, and only three or four (Hotmail, Yahoo, and Gmail and, it appears , AOL) are even starting to test using engagement.  Therefore, for a great majority of ISPs, seed-list-based monitoring still works as an extremely accurate proxy for your overall deliverability rates. 

We recently embarked on a research project to look at how statistically valid our seeds are by comparing them to real-life subscriber data pulled from a sample of about 2 million actual user mailboxes. We found that in 85% of the time, the delivery rates reported by the seed list and the actual delivery rates from these 2 million mailboxes are statistically the same.  In the cases where seeds and “panel” delivery rates did not match, it was due to extreme engagement rates.  We found outliers at both ends of the engagement spectrum – both very high and very low. But that was most commonly due to high levels of negative engagement, meaning that a majority of the subscribers were never opening emails, deleting emails without reading, never replying to messages and never marking the emails as “not spam” if found in the spam folder. 

So will subscriber data end up replacing seed lists for inbox placement monitoring? No. Seed lists are still very useful for determining inbox placement rates and doing root cause analysis on why emails are not being delivered to the inbox. But we will see the future of deliverability monitoring also include real-life subscriber data alongside seed lists.  This will help overcome the discrepancies for campaigns with very high negative engagement factors as well as for senders as a whole. 

Seed list monitoring will never be completely replaced by subscriber data.  Seeds give a better indication of how the filters at any ISP will deliver email for their general population as a whole.  Subscriber data will give insights into the small population of your list that are on the outside fringe of engagement.  When used together, they provide the best picture of where your emails are being delivered at the three ISPs that use engagement.