Steve Jenkins' Blog » Linux

How to Back Up a Linux Server to Amazon S3 with Duplicity and AutoMySQLBackup

Steve Jenkins — Sat, 05 Nov 2011 21:42:57 +0000

I often say there are only two types of data: data that is backed up, and data that is waiting to be lost.

Even if you’re the most contentious hax0r who always backs up your data, if your backups are stored in the same physical location as the source data, then your data is still ”waiting to be lost” in the event of a fire, flood, theft, or other disaster. By combining Amazon’s low-priced S3 (Simple Storage Solution) Cloud-based storage solution with some excellent some open source backup tools, you can now be more prepared than ever without spending a fortune.

This how-to demonstrates how I combined the following tools to automate my off-site backups:

Amazon S3: cheap, secure, redundant, off-site storage service
AutoMySQLBackup: free software to create backups of MySQL databases
Duplicity: free software that does smart backups to remote locations
GPG: allows encryption and signing of data for privacy
dt-s3-backup.sh: a slick shell script that ties all these tools together

Step 1: Set up your Amazon S3 Storage Bucket

I won’t walk through all the steps to do this, as Amazon makes it easy. Just sign up for their S3 service (you only pay for what you use), sign in, find the Security Credentials page and take note of your Access Key ID and your Secret Access Key. You’ll need them later. You should also set up an S3 Bucket to store your backups. Write down the name of your bucket for use in a later step.

Step 2: Download AutoMySQLBackup (optional)

If you don’t have any MySQL databases to back up, or you have your own preferred method of backup in your databases, you can skip this step. AutoMySQLBackup is a free utility that quickly and easily create dumps of your MySQL data – which we’ll back up to Amazon S3 in a later step.

Download AutoMySQLBackup from SourceForge and run the simple install.sh script to set it up. I followed this excellent blog post to help me get AutoMySQLBackup configured and working. I had to make a few minor changes because I’m using a more current version of AutoMySQLBackup and some of the variable names in the config file were different, but it’s pretty straightforward. Once you’ve got it backing up your databases, you’re ready to move on.

Step 3: Download and Install Duplicity

Duplicity is the program that does most of the heavy lifting in this situation. It manages the actual file backup (full or incremental), compression, encryption, and the file transfer to any number of off-site storage locations. Lots of documentation is available online, in case your needs differ from the ones explained here. As always, Google is your friend.

To install Duplicity if you’re running Fedora, RHEL, or CentOS, it’s as simple as doing:

yum install duplicity

For Ubuntu or Debian users, do:

apt-get duplicity

If you’re running some other flavor of Linux, refer to the Duplicity website for help installing.

Step 4: Create a GPG Key for Backups

Because you’re going to be transferring your precious data over the Internet, and storing it in an off-site location that shouldn’t be, but still technically could be, accessed by snooping Amazon employees or hackers, it’s best to encrypt your data before sending it… “to the CLOUD!” Seriously, those commercials are so annoying.

Even if you already have a GPG key, I recommend creating a separate one just for backups (which we’ll also store in a secure location later so you’re never stuck without the ability to decrypt your data later). Do:

gpg --gen-key

You can accept all the defaults, but make sure you use a passphrase when creating this key, since Duplicity will require it. After you’ve answered all the questions, the output should look something like this:

We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
.++++++++++++++++++++..+++++++++++++++++++++++++++++++++++++++++++++++++++++++.+++++
.+++++..+++++.+++++++++++++++++++++++++++++++++++++++++++++.....>.++++++++++................................+++++
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
+++++.++++++++++..++++++++++...++++++++++...+++++.+++++..+++++.+++++..+++++++++++++++.+++
++++++++++++..+++++++++++++++..++++++++++..+++++++++++++++++++++++++...+++++..+++++>+++
+++++++>.+++++>+++++......................+++++^^^
gpg: key 1F6C9247 marked as ultimately trusted
public and secret key created and signed.

gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   2  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 2u
pub   2048R/1F6C9247 2011-11-05
      Key fingerprint = FC81 D8E3 8090 EEE3 1D98  E000 045C D80E 1F6C 9247
uid                  Backup Key 
sub   2048R/12D6A5B0 2011-11-05

Take note of your key’s public GPG Key ID, which is listed on the line where it says “key xxxxxxxx marked as ultimately trusted” (in this example, it’s 1F6C9247). You can also find your GPG key’s public ID with:

gpg --list-keys

which will spit out something like:

pub   2048R/1F6C9247 2011-11-05
uid                  Backup Key 
sub   2048R/12D6A5B0 2011-11-05

You’ll see your key’s ID on the top row after the slash. Write it down (don’t worry, it’s not a security risk like a password) to refer to in the next step.

Step 5: Download and Configure dt-s3-backup Script

While trying to make all the aforementioned tools work together, a stumped across a very cool script that already did it for me. This blog post explains the script, and the script itself is hosted on GitHub.

Download the script to your server (I put mine in /usr/local/bin) and then open it up in an editor. You’ll need to put the following in the appropriate locations inside the script:

AWS_ACCESS_KEY_ID: Your Amazon Access Key (duh!)
AWS_SECRET_ACCESS_KEY: Your Amazon Secret Access Key (double duh!)
GPG_KEY: Your GPG Key ID of the key you created in the previous step
ROOT: I changed this to just “/” so that I could back up anything on the system. You’ll pick the exact directories you want in a bit.
DEST: Since we’re backing up to Amazon S3, comment out the “file:” line, uncomment the “s3+http:” line, and put the name of the Amazon S3 bucket you created for backups in the first step. If your bucket name were “my.awesome.backups” then this line would be DEST=”s3+http://my.awesome.backups/”

Skip the INCLIST and EXCLIST options for now, and tinker with the STATIC_OPTIONS to your liking. These will simply be passed to Duplicity, so you can check the Duplicity docs for all the possilibities. I have mine set to STATIC_OPTIONS=”–full-if-older-than 4W” which means my backup (which I run daily) will do incremental backups unless it’s been 4 weeks, in which case it will do a full backup. I also kept the default CLEAN_UP_TYPE and CLEAN_UP_VARIABLE settings. Again, refer to the Duplicity docs for other options.

Finally, I also tinkered with the Logfile settings and Email Alert settings.

Step 6: Choose which directories to include and exclude

Use the INCLIST and EXCLIST sections of the dt-s3-backup.sh script to list which directories you want to include and exclude while doing your backups. Examples are shown in the script. Make sure that whatever directory you used to store your database backups with AutoMySQLBackup is included. If you want hidden directories excluded, be sure to include them. The following are my lists:

INCLIST=(  "/www/" \
           "/etc/" \
           "/home/" \
           "/root/" \
           "/usr/local/bin/" \
           "/usr/local/backups/db/" \
        )

 EXCLIST=(   "/www/logs" \
            "/etc/selinux" \
            "/home/*/Download/" \
            "/root/*/Download/" \
            "/home/*/.*/" \
            "/root/.*/" \
            "/home/*/logs" \
            "/home/*/Maildir" "/home/*/mail" "/root/Maildir" "/root/mail" \
        )

These settings work for me, but there’s no guarantee they will work for you. It’s your data, so you should completely understand what is and isn’t going to be backed up.

Step 7: Do a Test Run

To test things out, find the following line in the dt-s3-backup.sh script and uncomment it (remove the #):

#ECHO=$(which echo)

As explained in the comments, this will run the script in test mode, which will spits out the full Duplicity command and send it to the email address you set up in the Email Alert settings.

Save your edited version of the script and run it with:

dt-s3-backup.sh --backup

Because it’s in test mode, it should think for a bit and then email you some output, which includes the full command that will be passed to Duplicity. If everything looks good, comment the #ECHO line out again, and go for it:

dt-s3-backup.sh --backup

Depending on many factors (the amount of data you’re backing up, the speed of your system, the speed of your connection to Amazon S3, the phase of the moon), you’ll have to wait for a bit. My system takes about 5 minutes to run a full backup.

If something goes wrong, check all your edits, and check the links to the other blog posts I’ve included. I won’t be any help answering support questions in this thread, because I’m not the author of any of these applications.

Step 8: Check Your Files

Assuming your backup worked, you can ask Duplicity to list all the files in your backup with:

dt-s3-backup.sh --list-current-files | more

Keep in mind that these will count as a requests against your Amazon S3 allowance. You get a bunch of free ones, but managing your Amazon bill is completely your responsibility.

Other options for dt-s3-backup.sh are available in its README file. I recommend experimenting with them until you’re familiar with the ones you’ll need.

Step 9: Automate

Once everything is working as you want it, don’t forget to create cron jobs for AutoMySQLBackup and dt-s3-backup.sh, I dump my databases nightly, and I do an incremental backup with dt-s3-backup weekly. Use whatever settings work best for you.

Step 10: Provide Feedback

I always welcome your feedback, especially if you have suggestions for making the process in this article easier to do or understand. If you have a different backup method that works for you, please feel free to share it. Because I’m not the author of any of these utilities, however, I can’t provide support in using them. Check the links I’ve provided for support, or contact the application authors directly if you’re having trouble.

Good luck moving your data from “waiting to be lost” to “backed up.” I know I sleep better knowing I’m better prepared to deal with disaster!

How To Use a Free StartSSL Certificate in Postfix for SSL/TLS

Steve Jenkins — Sat, 24 Sep 2011 02:59:35 +0000

Most of us use self-signed SSL certificates when setting up secure SMTP connections on our servers. And why not? It’s free! You do have to put up with your mail client “warning” you that the identity of the remote server can’t be verified, but that’s worth the minor inconvenience for the price. But… what if you could use an actual SSL certificate on your server, and get rid of those warnings, and still do it for free?

You can – thanks to StartSSL - the self-proclaimed “Swiss Army Knife of Digital Certificates & PKI.”

While the interface may not be as slick as other SSL providers (like GoDaddy or Network Solutions), what StartSSL lacks in design they make up for in value. You can actually get a 128/256-bit Encrypted Class 1 SSL/TLS + S.MIME certificate for free. Seriously, no strings attached.

Step 1: Sign up and Get Verified

The first step is to sign up at StartSSL and verify your personal identity via email, and also verify that you have admin authority for the domain. I noticed that Chrome won’t work properly with their authentication procedure, so you’ll need to use IE, Firefox, or Safari. I won’t go through all the verification steps here. Just follow the instructions on their website to get verified, and get to the Control Panel. Or, try their Express Lane option to get verified and receive a certificate in one step.

IMPORTANT: Make sure you choose the proper domain and subdomain name for your certificate when you come to that point. You can change your mind later, but you’ll have to pay $25 to do so. Just be sure you understand everything before you start clicking buttons…

Eventually, you’ll reach the Certificates Wizard. Select a Web Server SSL/TLS Certificate. Then press the Skip>> button, because you’ll want to create your own private key and certificate request on your server.

Step 2: Create Your Key and CSR

The next screen is where you submit your certificate request (CSR). But first, you’ll need to build your private key and the CSR itself. Make a directory to store all your SSL files with

mkdir -p /etc/ssl
cd /etc/ssl

Of course, you can choose any directory to store your SSL stuff, and you may already even have one. Use whatever directory you like. I personally like /etc/ssl because it’s easy to remember.

Inside your /etc/ssl directory, do:

openssl req -new -newkey rsa:2048 -nodes -keyout hostname.domain.key -out hostname.domain.csr

You can use any filename you want for the key and csr, but I like to include the subdomain and hostname of the mail server in mine, such as mail.example.key and mail.example.csr.

This command build both your private key and the CSR. Answer the questions (you don’t have to answer the optional ones), and be sure that when it requests the Common Name, use hostname and domain name of your server as you’ll be entering it in your email client (such as mail.example.com). If you make a mistake anywhere along the way, you can CTRL+C out of it, or just re-run the command to write a new key and CSR over the existing ones.

Once the command is finished, do an ls to see the two files you just created.

Your private key must be kept, well, private. So get into the habit of setting proper permissions for private keys right after you make them.

chmod 0640 hostname.domain.key

Step 3: Give your CSR to StartSSL and receive your certificate

Spit out the contents of your CSR with:

cat hostname.domain.csr

Then copy the entire contents of the file (including the:

-----BEGIN CERTIFICATE REQUEST-----

and

-----END CERTIFICATE REQUEST-----

lines, then paste it into the text area of the StartSSL CSR request page.

Hit the Continue>> button, and then select the specific subomain and domain you’ll use (such as mail.example.com or www.example.com) to access your server. For the free Class 1 certificate, you can include the domain and only one subdomain. This means the certificate will work for example.com and www.example.com, or example.com and mail.example.com.

If you want your certificate to work for two or more subdomains in addition to the primary domain, you’ll need to pay $59.99 to become Class 2 verified. But the good news is that you can then generate unlimited Class 2 certificates, which allow multiple and/or wildcard subdomains on your certificates. This is actually what I did, and it’s still a bargain.

After your request is processed, the certificate back from StartSSL, copy the contents and paste them into a file in your /etc/ssl directory called hostname.domain.crt (using your hostname and domain name, of course).

At this point, it’s a good idea to download all three files (key, csr, and crt) and store them somewhere safe. I have a secure USB key that I keep for such a purpose. Your .key file is irreplaceable at this point. If you lose it or accidentally delete it, you’ll have to revoke your certificate, which will cost you $25.

Step 4: Download the StartSSL Bundle

Certificates are about trust, and so you’ll need to inform your server that you trust StartSSL. List of certificate authorities that your server can trust are stored in Certificate Bundles. Download StartSSL’s bundle with:

wget --no-check-certificate https://www.startssl.com/certs/ca-bundle.pem -O startssl-ca-bundle.pem

If you have a pre-existing certificate bundle file somewhere on your server (such as ca-bundle.crt or cacert.pem), you may want to copy it into your /etc/ssl directory:

cp /etc/pki/tls/certs/ca-bundle.crt /etc/ssl

Then you’ll need to combine the StartSSL ca-bundle with your existing bundle (this step just copies the StartSSL bundle to the new filename if you didn’t have an existing bundle):

cat startssl-ca-bundle.pem >> ca-bundle.crt

Now edit your Postfix main.cf file and put in the updated locations of your private key, certificate, and bundle:

smtpd_tls_key_file = /etc/ssl/mail.scuderia.key
smtpd_tls_cert_file = /etc/ssl/mail.scuderia.crt
smtpd_tls_CAfile = /etc/ssl/ca-bundle.crt
smtp_tls_CAfile = $smtpd_tls_CAfile

This article assumes that you’ve probably already got Postfix set up for SSL/TLS with a self-signed certificate, so all the main.cf settings in addition to the ones I’ve mentioned above for TLS should already be in place. This article won’t explain how to do that (but there are plenty of great articles out there to get that working). I will, however, show you what the SMTP TLS section of my main.cf file looks like, in case it’s any help:

# SMTP TLS
smtpd_tls_auth_only = no
smtp_use_tls = yes
smtpd_use_tls = yes
smtp_tls_note_starttls_offer = yes
smtpd_tls_key_file = /etc/ssl/mail.domain.key
smtpd_tls_cert_file = /etc/ssl/mail.domain.crt
smtpd_tls_CAfile = /etc/ssl/cacert.pem
smtp_tls_CAfile = $smtpd_tls_CAfile
smtpd_tls_loglevel = 1
smtpd_tls_received_header = yes
smtpd_tls_session_cache_timeout = 3600s
tls_random_source = dev:/dev/urandom

Once you’ve got your settings right, restart Postfix with:

service postfix restart

Now you’re connecting to your outgoing mail server with a third-party signed certificate!

Git Commands I Like

Steve Jenkins — Wed, 21 Sep 2011 00:44:38 +0000

This is going to be one of those posts that is probably useless to everyone else, but valuable enough to me that I’ll read it over and over. These are the Git commands I use most when working with projects on my GitHub, and with Git repos to which I have read-only access.

(Big thanks to Murray K. and Todd L. for hand-holding me through this )

For my projects, I use a watered down version of the branching model presented in this article, so that I always have two branches:

Master: which always represents a release version of a project
Develop: which represents the progress I’m making toward the next release version

Therefore, for reasons that should be obvious, I always request that collaborators make pull requests only on the Develop branch.

Making changes to Develop

To check out my Develop branch, I do:

git checkout develop

I make whatever changes I want, and then make sure the files I want to include in commits are added with:

git add

(I only need to do this once per file.)

When I’m ready to commit at least one changed file, I do:

git commit -a

and then add notes about the changes I made.

Then I do:

git push

to push my local version of the Develop repo to GitHub.

Merging Develop into Master

Technically, with the branching model I’m using, I should never commit changes directly to Master. All updates should be put into Develop, and then once Develop represents a version that I think is ready to be released, and all the changes I want are committed to Develop and pushed, I merge Develop into Master with:

git checkout master
git merge develop
git push

Checkout Out Someone Else’s Repo and Submitting A Diff

When checking out someone else’s repo and submitting a diff to them, I do the following:

Do a git checkout of their develop branch with (usually):

git checkout develop
git pull

Then I create my own temporary branch from that with:

git checkout -b develop_branchname

Next, I edit or copy my changes into the branch and commit them with:

git commit -a

If I don’t have write/push access to the repo, I need to create a patch to send to the author with:

git format-patch develop..develop_branchname

After sending the diff, I can delete my temporary branch with:

git checkout develop
git branch -D develop_branchname

Simple CloudFront Invalidation of a Single File via HTTP

Steve Jenkins — Wed, 07 Sep 2011 06:05:38 +0000

I found a great post by Clay Loveless who wrote a PHP script to batch invalidate files files on an Amazon Web Services CloudFront distribution. You rock, Clay. Amazon doesn’t even let us do that (yet?) via the AWS Mangament Console.

Until now, when I wanted to manually invalidate a single file, I had to fire up CloudBerry S3 Explorer and scroll through thousands of files to get to the one I wanted to invalidate. For automated on-the-fly batch sync and CloudFront invalidation, I still rely on the excellent s3cmd command-line tool.

I’m not a developer, but I can lightly hack my way around PHP code – especially when it’s as clean as Clay’s. I took his excellent script one small step farther to allow invalidating a single file via HTTP (using your browser) by including the to-be-invalidated filename in a URL, with the ability to optionally pass the CloudFront Distribution ID via a separate variable in the URL (handy for those of us managing multiple distributions). I highly recommend making sure you place this file in a password protected area of your website.

So here’s my modified version of Clay’s PHP script providing a simple way to invalidate a single file on CloudFront using your browser:

/**
 * Super-simple AWS CloudFront Invalidation Script
 * Modified by Steve Jenkins  to invalidate a single file via URL.
 * 
 * Steps:
 * 1. Set your AWS Access Key
 * 2. Set your AWS Secret Key
 * 3. Set your CloudFront Distribution ID (or pass one via the URL with &dist)
 * 4. Put cf-invalidate.php in a web accessible and password protected directory
 * 5. Run it via: http://example.com/protected_dir/cf-invalidate.php?filename=FILENAME
 *    or http://example.com/cf-invalidate.php?filename=FILENAME&dist=DISTRIBUTION_ID
 * 
 * The author disclaims copyright to this source code.
 *
 * Details on what's happening here are in the CloudFront docs:
 * http://docs.amazonwebservices.com/AmazonCloudFront/latest/DeveloperGuide/Invalidation.html
 * 
 */

$onefile = $_GET['filename']; // You must include ?filename=FILENAME in your URL or this won't work

if (!isset($_GET['dist'])) {
        $distribution = 'DISTRIBUTION_ID'; // Your CloudFront Distribution ID, or pass one via &dist=
} else {
        $distribution = $_GET['dist'];
}

$access_key = 'AWS_ACCESS_KEY'; // Your AWS Access Key goes here
$secret_key = 'AWS_SECRET_KEY'; // Your AWS Secret Key goes here
$epoch = date('U');

$xml = <<
    {$onefile}
    {$distribution}{$epoch}
EOD;


/**
 * You probably don't need to change anything below here.
 */
$len = strlen($xml);
$date = gmdate('D, d M Y G:i:s T');
$sig = base64_encode(
    hash_hmac('sha1', $date, $secret_key, true)
);

$msg = "POST /2010-11-01/distribution/{$distribution}/invalidation HTTP/1.0\r\n";
$msg .= "Host: cloudfront.amazonaws.com\r\n";
$msg .= "Date: {$date}\r\n";
$msg .= "Content-Type: text/xml; charset=UTF-8\r\n";
$msg .= "Authorization: AWS {$access_key}:{$sig}\r\n";
$msg .= "Content-Length: {$len}\r\n\r\n";
$msg .= $xml;

$fp = fsockopen('ssl://cloudfront.amazonaws.com', 443, 
    $errno, $errstr, 30
);
if (!$fp) {
    die("Connection failed: {$errno} {$errstr}\n");
}
fwrite($fp, $msg);
$resp = '';
while(! feof($fp)) {
    $resp .= fgets($fp, 1024);
}
fclose($fp);
print ''.$resp.''; // Make the output more readable in your browser

            view raw
            cf-invalidate.php
            This Gist brought to you by GitHub.
          

SSH slow to connect? Turn UseDNS off

Steve Jenkins — Tue, 06 Sep 2011 18:16:27 +0000

Another quick tidbit I learned when one of my CentOS boxes was slow to connect via SSH. Connection happened quickly when I would connect from a box on the same router, but connecting from my home machine caused a delay of 20-30 seconds each time. The fix was uncommenting this line in /etc/ssh/sshd_config:

#UseDNS yes

and changing it to:

UseDNS no

I believe the delays were being caused by a combination of IPv6 and DNS, but regardless of the cause, that was the fix.

How to set up two NICs on different subnets with static routes and separate default gateways that remain after a reboot

Steve Jenkins — Mon, 05 Sep 2011 03:50:01 +0000

This is one of those blog posts that’s more written for my own benefit to refer to later than for the benefit of the Web-at-large, but if someone else happens to stumble upon it and it helps you out, then cool. I had to migrate a Fedora box (the one that hosts my blog, actually) from an old IP to a new IP on a different subnet, and I didn’t want any downtime during the DNS move. Luckily, my Fedora box has multiple NICs, but when I was using just the old IP address, I was only using one NIC (which was eth0). I wanted to give eth0 the new IP address, so first I edited /etc/sysconfig/network-scripts/ifcfg-eth0 to include to the new IP address & default gateway, then I set up /etc/sysconfig/network-scripts/ifcfg-eth1 to use to the existing IP address & gateway.

On my CentOS 5 boxes (which were also migrating to the new subnet) setting up the second NIC and plugging it in was enough to make everything Just Work. CentOS 5 seemed to be able to manage routes in and out of both IPs just fine. However, my Fedora 12 box was a bit more picky – even after I did a chkconfig NetworkManager off to stop NM from continually messing with my settings! I still don’t know why it doesn’t Just Work on Fedora, so if someone has an explanation for the difference, I’d love to hear about it in the comments. Anyway, Fedora can only have one default gateway, and it seemed to choose the one that was listed in the eth1 config file (I’m guessing it’s because that adapter comes up last). This resulted in me being able to connect to my box on the old IP on eth1, but not on the new IP on eth0.

I eventually found a solution by combining two excellent blog posts (this one and this one) as well as some trial and error in figuring out how to make the changes persist on a reboot. Of course, after the DNS migration is complete in a week or so, I’ll remove all this configuration because I’ll only have one IP and one gateway, which is why I wanted to write all this down in my blog just in case I ever need to do it again.

The first blog post explained how to set up a new policy routing table entry with:

# echo "1 new" >> /etc/iproute2/rt_tables

I called my policy routing table “new” because it was going to be applied for the new IP. The post then explained how to manually create two new route entries using the ip route add command, and also explained how to create two new rules entries in my “new” policy routing table using ip rule add. The problem is, however, that when I ran service network restart, the name of my “new” policy routing table would remain in /etc/iproute2/rt_tables (good) and the IP rules would also remain (also good), but both of the IP route commands would be lost (bad). And when I rebooted the machine, the policy routing table name “new” survived in /etc/iproute2/rt_tables (good), but all of the actual rules and route settings were lost (bad), making the machine unreachable again on the new IP (extremely bad).

The second blog post also discussed using the ip route add commands, but (kinda) explained how to make those settings persist by creating a /etc/sysconfig/network-scripts/route-eth0 file and placing my ip route add commands inside… minus the ip route add part. So my /etc/sysconfig/network-scripts/route-eth0 looks like this:

123.456.789.0/24 dev eth0 src 123.456.789.25 table new
default via 123.456.789.1 dev eth0 table new

This assumes that 123.456.789.0/24 is the network range on the new subnet I’m creating the route for, 123.456.789.25 is the new IP address on eth0, and 123.456.789.1 is the default gateway on the new subnet. Now, when the machine is rebooted or I run service network restart the ip route add commands will be executed.

But that still left the problem of making the rules survive a reboot, and neither blog post had any suggestions. However, since the rules persist on a service network restart, so only needed to make sure they were executed once every time the system booted – and that’s exactly what /etc/rc.d/rc.local is for. So I added these lines to my /etc/rc.d/rc.local:

# Temporary routing rules while two public NICs are active
ip rule add from xxx.xxx.xxx.25/32 table new
ip rule add to xxx.xxx.xxx.25/32 table new
ip route flush cache

If you want more details, the second blog post does a good job explaining these rules.

I tested everything with a reboot (both IPs on both NICs responded to pings) and then also ran another service network restart for good measure. Everything worked, so I could go ahead and begin the migration of the DNS.

As a somewhat related aside, Postfix’s smtp_bind_address command also came in really handy during this migration to force Postfix to send mail out the new IP while both NICs were active and I waited for the DNS to propagate. Again, that’s more for my future reference than anything, but it wasn’t worth dedicating a whole blog post to that idea, so I’ll slap it in here.

And that’s how you run two NICs on two different subnets with static routes and separate default gateways on Fedora!

Installing OpenDKIM RPM via Yum with Postfix or Sendmail (for RHEL / CentOS / Fedora)

Steve Jenkins — Tue, 16 Aug 2011 00:53:34 +0000

For those who want or need to compile and install OpenDKIM from the source code, you can follow the instructions I wrote in this article.

If you’re looking for the fastest and easiest way to get OpenDKIM running on a RedHat system, I currently maintain the OpenDKIM package in the Fedora and EPEL repositories. This article will help you install the RPMs and then configure OpenDKIM with Postfix or Sendmail.

For general information about DKIM, check out http://www.dkim.org/. For more information about the OpenDKIM project, check out http://www.opendkim.org/.

Before you start

This tutorial assumes the following:

You are running a “modern” RedHat-compatible Linux distro (RHEL 5/6, CentOS 5/6, Fedora, etc).
You are running a milter-aware MTA, such as Postfix 2.3.3 or newer (do postconf -d mail_version to check) or Sendmail.
Your Postfix or Sendmail configuration is currently working (this is very important – you don’t want to troubleshoot two programs at once).
If you’re using Postfix, Sendmail is turned off (do service sendmail status to verify).
If you’re using Sendmail, Postfix is turned off (do service postfix status to verify).
The necessary commands in this tutorial are done as root. If you don’t know what that means, then you probably shouldn’t be doing this. You may be able to get away with just using sudo, but I wanted to make sure I didn’t run into any path issues, so I do it as root.

Install OpenDKIM with Yum

If you’re running Fedora 14 (or newer) or RHEL/CentOS 5 (or newer) then you can use Yum to quickly install OpenDKIM (RHEL/CentOS users must have the EPEL repositories enabled). Just do:

yum install opendkim

This will download and install OpenDKIM with all the default configuration options included below.

For those who like getting their hands dirtier, you can manually download one of my RPMs or even build your own RPM from my Source RPM, which are all available through the Fedora BuildSystem.

Generate keys for signing

Now you’re getting to the fun part. You need to generate a private and a public key for each of the domains for which you wish to sign mail. The private key is stored away from prying eyes on your server, while the public key gets published in your domain’s DNS records so that receiving mail servers can verify your DKIM-signed mail.

To make things easy, the first time you start opendkim after installing the RPM package, it will generate a default set of keys in /etc/opendkim/keys/ using your server’s domain name and the selector name “default.” However, if you want to sign for any virtual hosts or choose a different selector name than then default, you can easily generate your own keys. It really only takes a few seconds. Or, if you’re happy with the default keys, you can move on to the next step.

If you’re really hard-core, you can always build the keys manually. Or, you can use the easy script included with OpenDKIM to do it for you. I’ve manually generated enough keys in my life, so I use the script.

Before running this script, decide now what the name of your selector is going to be. A selector is a unique keyword that is associated with both keys (public and private), included in all the signatures, and published in your DNS records. For simplicity, I use the word default as my default selector. Not very creative, but it’s effective. Feel free to choose something different, but if you do, you’ll need to use it consistently throughout your setup. Also, while this should go without saying, you should use your mail domain instead of example.com throughout the following steps.

Create your keys with:

mkdir /etc/opendkim/keys/example.com
/usr/bin/opendkim-genkey -D /etc/opendkim/keys/example.com/ -d example.com -s default
chown -R opendkim:opendkim /etc/opendkim/keys/example.com
mv /etc/opendkim/keys/example.com/default.private /etc/opendkim/keys/example.com/default

You can do a man opendkim-genkey if you’re interested in what additional options are available when creating your keys. In this example, I used the -D (directory) option, the -d (domain) option, and the -s (selector) options. That’s all you need to get this going.

Edit the configuration files

You’re getting really close now. You need to create and/or edit four files:

/etc/opendkim.conf – OpenDKIM’s main configuration file
/etc/opendkim/KeyTable – a list of keys available for signing
/etc/opendkim/SigningTable - a list of domains and accounts allowed to sign
/etc/opendkim/TrustedHosts – a list of servers to “trust” when signing or verifying

On install, the RPM package should have created a simple /etc/opendkim.conf file on your system. By default, this file is set up for verification only. In order to sign outgoing mail, you’ll have to comment, uncomment, and configure some additional options in the configuration file. Use your favorite text editor to open /etc/opendkim.conf and make it look like this:

## CONFIGURATION OPTIONS

# Specifies the path to the process ID file.
PidFile /var/run/opendkim/opendkim.pid

# Selects operating modes. Valid modes are s (signer) and v (verifier). Default is v.
Mode    sv

# Log activity to the system log.
Syslog  yes

# Log additional entries indicating successful signing or verification of messages.
SyslogSuccess yes

# If logging is enabled, include detailed logging about why or why not a message was
# signed or verified. This causes a large increase in the amount of log data generated
# for each message, so it should be limited to debugging use only.
#LogWhy yes

# Attempt to become the specified user before starting operations.
UserID  opendkim:opendkim

# Create a socket through which your MTA can communicate.
Socket  inet:8891@localhost

# Required to use local socket with MTAs that access the socket as a non-
# privileged user (e.g. Postfix)
Umask   002

# This specifies a file in which to store DKIM transaction statistics.
#Statistics              /var/spool/opendkim/stats.dat

## SIGNING OPTIONS

# Selects the canonicalization method(s) to be used when signing messages.
Canonicalization        relaxed/simple

# Domain(s) whose mail should be signed by this filter. Mail from other domains will
# be verified rather than being signed. Uncomment and use your domain name.
# This parameter is not required if a SigningTable is in use.
Domain                  example.com

# Defines the name of the selector to be used when signing messages.
Selector                default

# Gives the location of a private key to be used for signing ALL messages.
#KeyFile                 /etc/opendkim/keys/default.private

# Gives the location of a file mapping key names to signing keys. In simple terms,
# this tells OpenDKIM where to find your keys. If present, overrides any KeyFile
# setting in the configuration file.
KeyTable                 refile:/etc/opendkim/KeyTable

# Defines a table used to select one or more signatures to apply to a message based
# on the address found in the From: header field. In simple terms, this tells
# OpenDKIM how to use your keys.
SigningTable                 refile:/etc/opendkim/SigningTable

# Identifies a set of "external" hosts that may send mail through the server as one
# of the signing domains without credentials as such.
ExternalIgnoreList      refile:/etc/opendkim/TrustedHosts

# Identifies a set internal hosts whose mail should be signed rather than verified.
InternalHosts           refile:/etc/opendkim/TrustedHosts

You can do man opendkim.conf for more information on each of the options in this file.

Uncomment the Domain option (and include your actual domain name), the KeyTable, SigningTable, ExternalIgnoreList, and InternalHosts options. Also, since you’ll be using a KeyTable, you can comment the KeyFile option.

Next, you’ll need to create the three text files that you just uncommented in your config file. First, using your favorite text editor, create an /etc/opendkim/KeyTable file that looks like this:

default._domainkey.example.com example.com:default:/etc/opendkim/keys/example.com/default

The KeyTable file tells OpenDKIM where to find your keys. Each entry in the KeyTable file is a single line for each key location (for example, all of the text in the above example should be on a single line in your file). If you’re going to use multiple keys (to sign mail for virtual domains with different keys, for example), you’ll need to create a separate line in the KeyTable file for each domain, like this:

default._domainkey.example.com example.com:default:/etc/opendkim/keys/example.com/default
default._domainkey.example2.com example2.com:default:/etc/opendkim/keys/example2.com/default

Next, you need to create or edit the /etc/opendkim/SigningTable file. A default version of this file should have been installed in /etc/opendkim when you installed the RPM, so just uncomment the following line (or edit the file to include this line) so it reads:

*@example.com default._domainkey.example.com

The SigningTable file tells OpenDKIM how to use your keys, as in which senders should use which selectors for their signatures. In the above example, I’m saying that everyone (*) sending mail from the server “example.com” should use the selector named “default.” Again, for multiple domains and/or users, you’ll need multiple lines, like this:

*@example.com default._domainkey.example.com
bob@example2.com default._domainkey.example2.com
doug@example2.com default._domainkey.example2.com

In that example, everyone (*) sending mail from the server “example.com” can sign mail and should use the selector named “default.” But only Bob and Doug can sign mail for “example2.com” (also using a selector named default). It’s important to note that the * wildcard symbol will only work if the SigningTable option uses the refile: prefix before the filename (see the opendkim.conf documentation for more details).

Next, create an /etc/opendkim/TrustedHosts file that looks like this:

127.0.0.1
hostname1.example1.com
hostname2.example1.com
example1.com
hostname1.example2.com
hostname2.example2.com
example2.com

The TrustedHosts file tells OpenDKIM who to let use your keys. Because it’s referenced by the ExternalIgnoreList directive in your conf file, OpenDKIM will ignore this list of hosts when verifying incoming mail. And, because it’s also referenced by the InternalHosts directive, this same list of hosts will be considered “internal,” and OpenDKIM will sign their outgoing mail.

IMPORTANT: Make sure you list the IP address for localhost (127.0.0.1) in the TrustedHosts file or OpenDKIM won’t sign mail sent from this server. If you have multiple servers on the same network that relay mail through this server and you want to sign their mail as well, they must be listed in the TrustedHosts file. Put each entry on its own line. An entry can be a hostname, domain name (e.g. “example.com”), IP address, an IPv6 address (including an IPv4 mapped address), or a CIDR-style IP specification (e.g. “192.168.1.0/24″).

It should also go without saying (but I’ll say it anyway) that if you’re planning to sign outgoing mail for remote hosts, your mail server should have been previously configured to allow relaying for those hosts.

Edit your MTA configuration

Now you’re ready to tell your MTA about OpenDKIM.

Postfix Users:

Telling Postfix about OpenDKIM is easy. Just add the following lines to your Postfix main.cf file:

smtpd_milters           = inet:127.0.0.1:8891
non_smtpd_milters       = $smtpd_milters
milter_default_action   = accept

If you’re running a version of Postfix prior to 2.6, you may need to add:

milter_protocol   = 2

See http://www.postfix.org/MILTER_README.html#version for more info.

Don’t restart Postfix yet! You need to have OpenDKIM running first, or you’ll get errors in your maillog.

Sendmail Users:

Edit the .mc configuration file that was used to build your current sendmail.cf file. Add the following line:

INPUT_MAIL_FILTER(`opendkim', `S=inet:8891@localhost')

Then build and install a new sendmail.cf. If you don’t know how to build and install a sendmail.cf file, a quick Web search should shove you in the right direction. Explaining how to do that is beyond the scope of these instructions. I will, however, remind you that backing up your current sendmail.cf file is a good idea before you attempt any modifications.

Start OpenDKIM and restart your MTA

It’s time to fire things up! Assuming you’re using bash, do:

hash -r

to rehash your shell so you can find the init script.

Now start OpenDKIM with:

service opendkim start

You should get a message that says:

Starting OpenDKIM Milter:     [  OK  ]

However, if you get an error message such as:

Starting OpenDKIM Milter: opendkim: /etc/opendkim.conf: configuration error at line 6: unrecognized parameter

don’t freak out. You probably just mistyped something in one of the config files. Go to the line number of the file listed, and check your work against the example(s) in this article. Then try starting up OpenDKIM again.

Once it starts, Postfix users should refresh Postfix with:

postfix reload

and Sendmail users should do:

service sendmail restart

If everything looks good, I recommend running chkconfig on OpenDKIM to make sure it starts when you boot your server:

chkconfig opendkim on

If things didn’t go right, try some of these startup troubleshooting tips before moving on.

Startup troubleshooting tips

Tip 1: The best advice I can give when troubleshooting any mail issues (including OpenDKIM) is to start a second shell session in another window and do:

tail -f /var/log/maillog

while you’re starting, stopping, and/or restarting OpenDKIM and your MTA. This allows you to see more details about any errors in your configuration.

Tip 2: To get the most verbose information from OpenDKIM, make sure the LogWhy option in your /etc/opendkim.conf file is uncommented and set to Yes. If your outgoing mail isn’t getting signed and you want to know why, this should tell you.

Tip 3: If you can’t get things working on your own, I recommend subscribing to the OpenDKIM-Users discussion list at http://lists.opendkim.org/. It’s a low-traffic list with very helpful and friendly members (including me!) who are happy to nudge you in the right direction.

Adding DNS Records

Now that your mail server is signing outgoing mail and verifying incoming mail, you’ll need to put some information in your DNS records to tell other mail servers how your keys are set up, and provide the public key for them to check that your mail is properly signed. Do:

cat /etc/opendkim/keys/example.com/default.txt

The output should look something like this:

default._domainkey IN TXT "v=DKIM1; g=*; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDHY7Zl+n3SUldTYRUEU1BErHkKN0Ya52gazp1R7FA7vN5RddPxW/sO9JVRLiWg6iAE4hxBp42YKfxOwEnxPADbBuiELKZ2ddxo2aDFAb9U/lp47k45u5i2T1AlEBeurUbdKh7Nypq4lLMXC2FHhezK33BuYR+3L7jxVj7FATylhwIDAQAB" ; ----- DKIM default for example.com

If you manage your own DNS or have full access to your domain’s zone file, you’ll need to paste the entire contents of the default.txt file at the bottom of your domain’s zone file. If you’re using a web interface to manage your zone file, be careful that the long lines of the public key don’t wrap and create line-feed characters (or fix them if they do). Otherwise, your public key won’t work.

If you’re using GoDaddy’s Total DNS, the TXT Name would default._domainkey and the TXT Value would be everything inside the quotes (starting with v=). You can ignore the semi-colon and comments at the end.

If you’re using some other third-party DNS provider, follow their instructions for adding a new TXT Record.

You should also add another TXT Record to your zone file that reads:

_adsp._domainkey.example.com    IN    TXT    "dkim=unknown"

This record publishes your Author Domain Signing Practices. “Unknown” is the least strict setting, and the best place to start. You can learn more and tinker with other options later, but most people just use “Unknown” for now, since ADSP is relatively new (as of the writing of this post).

And, as long as you’re messing with your domain’s zone file, now might be a good time to ensure that you already have a valid SPF Record in place. Having both DKIM and SPF in place will increase your chances of having your outgoing mail successfully delivered.

Testing Things Out

As I mentioned in my troubleshooting tips, the best way to see that everything is working on the server side is to keep an eye on your /var/log/maillog file. Do a:

tail -f /var/log/maillog

When OpenDKIM starts (or restarts), you should see lines like:

opendkim[4397]: OpenDKIM Filter: mi_stop=1
opendkim[4397]: OpenDKIM Filter v2.4.2 terminating with status 0, errno = 0
opendkim[27444]: OpenDKIM Filter v2.4.2 starting (args: -x /etc/opendkim.conf)

When you send a mail that gets successfully signed, you should see:

opendkim[22254]: 53D0314803B: DKIM-Signature header added

The best way to check that your signed mail is being authenticated and that your DNS records are properly set up is to use one of the free testing services. My favorites are:

Brandon Checketts Email Validator
Send a signed email to: autorespond+dkim@dk.elandsys.com
Send a signed email to: sa-test@sendmail.net
Send a signed email to: check-auth@verifier.port25.com
(you can put all of the test email addresses in the To: field of a single outgoing message to test)

Each of these will tell you if things are working properly, and give you some pointers on troubleshooting if needed.

If you have a Gmail account, you can also send a signed message there for a quick and easy test. address Here’s what a signed message in Gmail will look like:

Look, Ma! My emails have DKIM Signatures!

The signed by: line tells you that the message has been verified as signed by the sender (you may need to press the show details link near the top of the message to see it). I like to click the Show Original link (under the Reply drop-down on the right) to see the signed headers in all their glory.

How to Install APC (Alternative PHP Cache) on CentOS 5.6

Steve Jenkins — Tue, 02 Aug 2011 03:18:12 +0000

There’s a lot of conflicting information out there on how to install the APC opcode cache on a CentOS 5.6 box. Here’s how I did it:

This tutorial assumes you’re running CentOS 5.6 on a dedicated server, and that you have superuser (root) access. These instructions may also worked in a VPS or shared hosting environment, but if you run into trouble, you should contact your provider’s tech support to see if they have any alternative steps you should take.

First, you need to make sure the following packages are installed:

yum install php-devel pcre-devel

Because you’re going to be compiling a package, you’ll need a C compiler (like gcc) and make. The easiest way to install all the right development tools is with:

yum groupinstall "Development Tools"

Be sure to include those quotes.

Now you’re ready to download and untar the package:

cd /usr/local/src
wget http://pecl.php.net/get/APC-3.1.9.tgz
tar -zxvf APC-3.1.9.tgz

Now it’s time to set it up!

cd APC-3.1.9
phpize
whereis php-config

Take note of where php-config is located (it’s usually in /usr/bin/php-config) as you’ll need it in the next step. If it’s in a different location, use that for the php-config path below:

./configure --enable-apc --enable-apc-mmap --with-apxs --with-php-config=/usr/bin/php-config
make
make install

Next, if you have an /etc/php.d/ directory on your server, create a file called /etc/php.d/apc.ini to store all your configuration settings (if it already exists, you can just edit the existing one). If you don’t have an /etc/php.d/ directory, you can add these configuration settings to your existing php.ini file, which is usually located at /etc/php.ini.

Here’s an example of my apc.ini file, with lots of comments for some guidance. This is a pretty straightforward configuration, so feel free to copy and paste:

; Enable the extension module
extension = apc.so

; Options for the APC module version >= 3.1.3
; See http://www.php.net/manual/en/apc.configuration.php

; This can be set to 0 to disable APC.
apc.enabled=1
; The number of shared memory segments to allocate for the compiler cache.
apc.shm_segments=1
; The size of each shared memory segment, with M/G suffixe
apc.shm_size=64M
; A "hint" about the number of distinct source files that will be included or
; requested on your web server. Set to zero or omit if you're not sure;
apc.num_files_hint=1024
; Just like num_files_hint, a "hint" about the number of distinct user cache
; variables to store.  Set to zero or omit if you're not sure;
apc.user_entries_hint=4096
; The number of seconds a cache entry is allowed to idle in a slot in case this
; cache entry slot is needed by another entry.
apc.ttl=7200
; use the SAPI request start time for TTL
apc.use_request_time=1
; The number of seconds a user cache entry is allowed to idle in a slot in case
; this cache entry slot is needed by another entry.
apc.user_ttl=7200
; The number of seconds that a cache entry may remain on the garbage-collection list.
apc.gc_ttl=3600
; On by default, but can be set to off and used in conjunction with positive
; apc.filters so that files are only cached if matched by a positive filter.
apc.cache_by_default=1
; A comma-separated list of POSIX extended regular expressions.
apc.filters
; The mktemp-style file_mask to pass to the mmap module
apc.mmap_file_mask=/tmp/apc.XXXXXX
; This file_update_protection setting puts a delay on caching brand new files.
apc.file_update_protection=2
; Setting this enables APC for the CLI version of PHP (Mostly for testing and debugging).
apc.enable_cli=0
; Prevents large files from being cached
apc.max_file_size=1M
; Whether to stat the main script file and the fullpath includes.
apc.stat=1
; Vertification with ctime will avoid problems caused by programs such as svn or rsync by making
; sure inodes havn't changed since the last stat. APC will normally only check mtime.
apc.stat_ctime=0
; Whether to canonicalize paths in stat=0 mode or fall back to stat behaviour
apc.canonicalize=0
; With write_lock enabled, only one process at a time will try to compile an
; uncached script while the other processes will run uncached
apc.write_lock=1
; Logs any scripts that were automatically excluded from being cached due to early/late binding issues.
apc.report_autofilter=0
;This setting is deprecated, and replaced with apc.write_lock, so let's set it to zero.
apc.slam_defense=0

The final step is to restart Apache with either:

service httpd restart

apachectl restart

To make sure APC is running, create a php file somewhere in your web root that simply contains:

then load the file and scroll down to the APC section to verify that it’s enabled.

For some cool statistics, copy the usr/local/src/APC-3.1.9/apc.php file somewhere in your web root, and then open it in your browser. Edit the file on the server if you’d like to add a password and enable login to see more details. Here’s a peek at what it looks like:

Congratulations! You’ve just installed APC on CentOS 5.6!

Bricked WRT54G with Blinking Power Light. Hard reset? TFTP? Nope – New Power Adapter.

Steve Jenkins — Wed, 20 Jul 2011 15:58:08 +0000

Like many owners of the ubiquitous Linksys WRT54G wireless router, I like to run aftermarket firmware on my device to unlock more of this router’s potential. I use one as my primary wireless router in my house, but I also use some as wireless Ethernet client bridges and wireless repeaters. So when I saw a WRT54G v3 for sale near me for $40 on Craiglist, I couldn’t pass up the opportunity to grab another one. I offered $30 and the seller accepted.

When I picked up the unit, the first thing I noticed was that the previous owner had upgraded the antenna to the +7db versions. Score! My $30 outlay was looking like a better deal by the minute. I also noticed that the power supply was not Linksys branded, and the output rating on the adapter was 9V 300mA. Once I got home, I plugged it in to test, and the unit lit up. But then the power light started flashing… and then kept flashing. I pulled the plug, did a hard reset, and tried again. Suck! Was this thing bricked already?

I tried all the standard tricks to unbrick this router, including everything on Recover from a Bad Flash on the DD-WRT website and the article I’d used successfully in the past: Bricked! Or, How to Resurrect a Dead Linksys WRT54G on WiFi Planet.

I was able to get my router to ping, using a combination of a 30/30/30 reset. I was also able to get a TFTP upload to complete successfully, using the original Linksys firmware, but I still couldn’t get the unit to boot or display the GUI. I also used TFTP to upload DD-WRT mini and Tomato, but still had the same results. I hate to admit that I even tried jumping pins 15 and 16. But even that didn’t work.

I was about to toss the unit in the trash (keeping the upgraded antenna, of course), but then remembered that the power supply wasn’t stock. There’s some discussion as to what the correct power settings are for these units, as they were designed to accept a range of adapters. I’ve heard anywhere from 5V to 12V will work. So I figured since I was going to toss the router anyway, messing with the power couldn’t do much damage. So I hit The Shack and purchased an adjustable wall wart that was capable of anywhere from 3-12V and 1000mA (otherwise known as 1 amp). I brought it home, plugged it in, and…

Everything worked. I had spent hours on resets, ping attempts, firmware upload attempts, taking it apart and putting it back together, all in vain. Power was the culprit. The unit must have been getting enough juice to turn on, and even ping, but eventually it would reset, flash all the lights, and attempt to boot up again – over, and over, and over. Giving it 12V and allowing it to draw up to 1 amp was the key. I uploaded the latest Tomato firmware, and now the unit runs like a champ!

So if your router is WRT54G is acting bricked, but nothing else seems to be working, maybe you should try the easy solution first: just swap out the power supply!

Fixing Postfix “certificate verification failed for gmail untrusted issuer” Error Message

Steve Jenkins — Tue, 07 Jun 2011 16:07:21 +0000

A while back, I noticed a recurring error in my maillog that read:

postfix/smtp: certificate verification failed for gmail-smtp-in.l.google.com[74.125.53.27]:25: untrusted issuer /C=US/O=Equifax/OU=Equifax Secure Certificate Authority

If you’re seeing this too, there’s an easy fix.

The problem stems from the fact that Google changed certificate providers from Thawte to Equifax, and your mail system doesn’t recognize the Equifax certificate authority as valid. The solution is to add a copy of the Equifax certificate to Postfix’s local root certificate store. And while we’re at it, we may as well add Thawte’s as well.

Step 1: Back up your original root certificate store

Before messing with any of the settings, do the following to backup your original root certificate:

# cd /etc/postfix/ssl
# cp cacert.pem cacert.pem.bak

Step 2: Create local copies of Equifax and Thawte certificates

Using your favorite text editor, create a file called Equifax_Secure_CA.pem in the /etc/postfix/ssl directory. Paste the following into the file:

Next, create a file called Thawte_Premium_Server_CA.pem in the /etc/postfix/ssl directory and paste the following into that file:

Step 3: Add the Equifax and Thawte certificates into your local root certificate file

Add the two new certificates into your local root certificate file with:

cat /etc/postfix/ssl/Equifax_Secure_CA.pem >> /etc/postfix/ssl/cacert.pem
echo >> /etc/postfix/ssl/cacert.pem
cat /etc/postfix/ssl/Thawte_Premium_Server_CA.pem >> /etc/postfix/ssl/cacert.pem

(the second command above adds a line break between the two certs in your local root cert file)

Step 4: Restart Postfix

Restart Postfix with:

# service postfix restart

Send a message through your SMTP server to a Gmail test address and you should no longer see those errors in your maillog!