This is one of those blog posts that’s more written for my own benefit to refer to later than for the benefit of the Web-at-large, but if someone else happens to stumble upon it and it helps you out, then cool. 🙂 I had to migrate a Fedora box (the one that hosts my blog, actually) from an old IP to a new IP on a different subnet, and I didn’t want any downtime during the DNS move. Luckily, my Fedora box has multiple NICs, but when I was using just the old IP address, I was only using one NIC (which was eth0). I wanted to give eth0 the new IP address, so first I edited /etc/sysconfig/network-scripts/ifcfg-eth0 to include to the new IP address & default gateway, then I set up /etc/sysconfig/network-scripts/ifcfg-eth1 to use to the existing IP address & gateway.

On my CentOS 5 boxes (which were also migrating to the new subnet) setting up the second NIC and plugging it in was enough to make everything Just Work. CentOS 5 seemed to be able to manage routes in and out of both IPs just fine. However, my Fedora 12 box was a bit more picky – even after I did a chkconfig NetworkManager off to stop NM from continually messing with my settings! I still don’t know why it doesn’t Just Work on Fedora, so if someone has an explanation for the difference, I’d love to hear about it in the comments. Anyway, Fedora can only have one default gateway, and it seemed to choose the one that was listed in the eth1 config file (I’m guessing it’s because that adapter comes up last). This resulted in me being able to connect to my box on the old IP on eth1, but not on the new IP on eth0.

I eventually found a solution by combining two excellent blog posts (this one and this one) as well as some trial and error in figuring out how to make the changes persist on a reboot. Of course, after the DNS migration is complete in a week or so, I’ll remove all this configuration because I’ll only have one IP and one gateway, which is why I wanted to write all this down in my blog just in case I ever need to do it again. 🙂

The first blog post explained how to set up a new policy routing table entry with:

echo "1 new" >> /etc/iproute2/rt_tables

I called my policy routing table “new” because it was going to be applied for the new IP. The post then explained how to manually create two new route entries using the ip route add command, and also explained how to create two new rules entries in my “new” policy routing table using ip rule add. The problem is, however, that when I ran service network restart, the name of my “new” policy routing table would remain in /etc/iproute2/rt_tables (good) and the IP rules would also remain (also good), but both of the IP route commands would be lost (bad). And when I rebooted the machine, the policy routing table name “new” survived in /etc/iproute2/rt_tables (good), but all of the actual rules and route settings were lost (bad), making the machine unreachable again on the new IP (extremely bad).

The second blog post also discussed using the ip route add commands, but (kinda) explained how to make those settings persist by creating a /etc/sysconfig/network-scripts/route-eth0 file and placing my ip route add commands inside… minus the ip route add part. So my /etc/sysconfig/network-scripts/route-eth0 looks like this:

123.456.789.0/24 dev eth0 src 123.456.789.25 table new
default via 123.456.789.1 dev eth0 table new

This assumes that 123.456.789.0/24 is the network range on the new subnet I’m creating the route for, 123.456.789.25 is the new IP address on eth0, and 123.456.789.1 is the default gateway on the new subnet. Now, when the machine is rebooted or I run service network restart the ip route add commands will be executed.

But that still left the problem of making the rules survive a reboot, and neither blog post had any suggestions. However, since the rules persist on a service network restart, so only needed to make sure they were executed once every time the system booted – and that’s exactly what /etc/rc.d/rc.local is for. So I added these lines to my /etc/rc.d/rc.local:

# Temporary routing rules while two public NICs are active
ip rule add from xxx.xxx.xxx.25/32 table new
ip rule add to xxx.xxx.xxx.25/32 table new
ip route flush cache

If you want more details, the second blog post does a good job explaining these rules.

I tested everything with a reboot (both IPs on both NICs responded to pings) and then also ran another service network restart for good measure. Everything worked, so I could go ahead and begin the migration of the DNS.

As a somewhat related aside, Postfix’s smtp_bind_address command also came in really handy during this migration to force Postfix to send mail out the new IP while both NICs were active and I waited for the DNS to propagate. Again, that’s more for my future reference than anything, but it wasn’t worth dedicating a whole blog post to that idea, so I’ll slap it in here.

And that’s how you run two NICs on two different subnets with static routes and separate default gateways on Fedora!