How-To: Postfix configuration to reduce Yahoo temporary deferrals using Transport Maps

If you’re getting “temporarily deferred” Postfix log entries when trying to send mail to Yahoo! that look like this:

host mta7.am0.yahoodns.net[66.196.118.34] refused to talk to me: 421 4.7.0 [GL01] Message from (xxx.xx.xx.xxx) temporarily deferred - 4.16.50. Please refer to http://postmaster.yahoo.com/errors/postmaster-21.html

status=deferred (host mta7.am0.yahoodns.net[66.196.118.35] said: 451 mta1336.mail.bf1.yahoo.com Resources temporarily unavailable. Please try again later [#4.16.1].

then you’re not alone. Yahoo’s mail servers have a reputation among mail server admins of being, well, downright finicky when it comes to accepting large volumes of mail. If you’re sending a large volume of legitimate mail (such as a newsletter), Yahoo is very picky about how much it will allow through at full speed before spitting out 421 “Yahoo temporarily deferred” error messages, or 451 “Yahoo resources temporarily unavailable”, or greylisting, or otherwise temporariliy blocking your Postfix server from sending mail to Yahoo mail recipients.

Reading the suggestions at http://postmaster.yahoo.com/ is a good start. Before doing any steps in this how-to, I recommend making sure your Postfix configuration is fully compliant with Yahoo’s suggestions (including the use of DKIM). But once you’ve done all that, if you’re still getting deferrals, then maybe you could benefit from setting up a custom transport for Yahoo (and any other stubborn mailers) using the transport_maps, _initial_destination_concurrency, _destination_concurrency_limit, _destination_rate_delay, and _destination_recipient_limit settings in Postfix.

How it Works

First, I have to admit that sometimes, the inner workings of Postfix are voodoo to me. I’m not a programmer, and the Postfix documentation is completely geared toward programmers. However, after trudging through Postfix mailing list archives and experimenting a bit, this is how I understand it. Postfix is highly efficient, and pretty smart in its default configuration. It tries to deliver as much of your outbound mail as possible as quickly as possible. But with a stubborn MTA like Yahoo (and sometimes Hotmail), you need to force Postfix to slow down its uber-efficient approach.

So, using transport maps, you can tell Postfix to watch out for emails to specific domains (like Yahoo) and treat them a bit differently.

Step 1: Setting up the Transport Maps

The first step is to determine which domains you want to treat differently. Obviously, in this example, we’re trying to set up something to eliminate (or at least reduce) Yahoo’s deferrals. So edit the /etc/postfix/transport file and create some maps that tell Postfix exactly which email domains are going to get the special “Yahoo” treatment. The email domain goes on the left, and the name of your custom transport goes on the right (always followed by a colon). The most basic approach would be to specifically list all the Yahoo email domains you want to cover in your /etc/postfix/transport file like this:

# Yahoo (USA)
yahoo.com       yahoo:
ymail.com       yahoo:
rocketmail.com  yahoo:

# Yahoo (INTL)
yahoo.ae        yahoo:
yahoo.at        yahoo:
yahoo.be        yahoo:
yahoo.ca        yahoo:
yahoo.ch        yahoo:
yahoo.cn        yahoo:
yahoo.co.il     yahoo:
yahoo.co.in     yahoo:
yahoo.co.jp     yahoo:
yahoo.co.kr     yahoo:
yahoo.co.nz     yahoo:
yahoo.co.th     yahoo:
yahoo.co.uk     yahoo:
yahoo.co.za     yahoo:
yahoo.com.ar    yahoo:
yahoo.com.au    yahoo:
yahoo.com.br    yahoo:
yahoo.com.cn    yahoo:
yahoo.com.hk    yahoo:
yahoo.com.mx    yahoo:
yahoo.com.my    yahoo:
yahoo.com.ph    yahoo:
yahoo.com.sg    yahoo:
yahoo.com.tr    yahoo:
yahoo.com.tw    yahoo:
yahoo.com.vn    yahoo:
yahoo.cz        yahoo:
yahoo.de        yahoo:
yahoo.dk        yahoo:
yahoo.en        yahoo:
yahoo.es        yahoo:
yahoo.fi        yahoo:
yahoo.fr        yahoo:
yahoo.gr        yahoo:
yahoo.ie        yahoo:
yahoo.it        yahoo:
yahoo.nl        yahoo:
yahoo.no        yahoo:
yahoo.pl        yahoo:
yahoo.pt        yahoo:
yahoo.ro        yahoo:
yahoo.ru        yahoo:
yahoo.se        yahoo:

However, listing all those domains forces you to stay up to date with any new domains that Yahoo might launch. So a smarter approach would be to two transport maps: one that’s a regular hash table, and another with a regular expression that simply catches any domain that starts with yahoo.

First, put ymail.com and rocketmail.com in your /etc/postfix/transport file, like this:

# Yahoo
ymail.com       yahoo:
rocketmail.com  yahoo:

Once you’re done editing the /etc/postfix/transport file (and after every edit from now on), remember to do:

postmap /etc/postfix/transport

to build the transport.db file.

Next, create a file called /etc/postfix/transport.regexp that looks like this:

# Yahoo Wildcards
/yahoo(\.[a-z]{2,3}){1,2}$/  yahoo:

That will catch all “yahoo dot anything” domains. Note that you don’t need to run postmap on regular expression tables, so now you’re ready to tell Postfix how to read your transports.

Step 2: Include the New Custom Transports in master.cf

As always, before messing with /etc/postfix/master.cf, make a backup. Then add the following lines at the bottom:

yahoo     unix  -       -       n       -       -       smtp
        -o syslog_name=postfix-yahoo

This tells Postfix that the transport called “yahoo” gets handed off to the Postfix smtp process, and the -o syslog_name option tags the use of this transport in the mail log so I easily tell when this transport is used.

Step 3: Create Custom Settings in main.cf

The third step of the process is to create some custom settings in your main.cf file to tell Postfix exactly what to do differently when it encounters an outbound mail domain that matches your transport maps. Back up your /etc/postfix/main.cf file, then add these lines:

transport_maps = hash:/etc/postfix/transport, regexp:/etc/postfix/transport.regexp

yahoo_initial_destination_concurrency = 1
yahoo_destination_concurrency_limit = 4
yahoo_destination_recipient_limit = 2
yahoo_destination_rate_delay = 1s

This tells Postfix to check your /etc/postfix/transport and /etc/postfix/transport.regexp files to look up which domains you’ve mapped to which transport, then it sets four specific configurations for the “yahoo” transport:

yahoo_initial_destination_concurrency = 1 will start out slowly by only sending one message per SMTP connection to a Yahoo’s MTA.
yahoo_destination_concurrency_limit = 4 after starting out slowly with just 1 message, Postfix will increase to allow up to four messages per SMTP connection to a Yahoo MTA.
yahoo_destination_recipient_limit = 2 will send the same message to no more than 2 recipients at a time
yahoo_destination_rate_delay = 1s will add a 1 second delay between the messages

This is where the Postfix voodoo kicks in for me, so feel free to experiment with these settings and tweak to your liking. The destination concurrency limit and the rate delay are the two you’ll probably want to tinker to keep Yahoo happy. Depending on your mailer reputation, they’ll be more strict or more relaxed on what they’ll allow for these two settings. The above settings that happen to work for my needs, but I still tweak them to experiment, and if you have a configuration that works well with Yahoo (or if you have other custom transports that help increase delivery), please feel free to share them in the comments.

Step 4: Restart Postfix and Test

Now you’re ready to try things out. Start the Postfix configuration with:

service postfix restart

You can’t just do a postfix reload, because changes to the master.cf require a full restart. Finally, do a tail -f on your maillog. On my CentOS system, that’s:

 tail -f /var/log/maillog

Now send a message to a Yahoo test account (I’m assuming you have an @yahoo.com test account) from or through your Postfix server. If everything worked right, you should see log entries that start with the date, local hostname, and then say postfix-yahoo/smtp. These are the messages that are being diverted through your new transport!

After using these settings for a few mailings, I’ve seen a drastic reduction in the amount of time it takes to deliver tens of thousands of opt-in email to Yahoo recipients. Hopefully, they’ll work for you, too!

Your feedback and comments are welcome below, and I’m especially interested to hear of any other custom transports you may be using, as well as your experiences with different settings for Yahoo.