Another quick tidbit I learned when one of my CentOS boxes was slow to connect via SSH. Connection happened quickly when I would connect from a box on the same router, but connecting from my home machine caused a delay of 20-30 seconds each time. The fix was uncommenting this line in /etc/ssh/sshd_config:

#UseDNS yes

and changing it to:

UseDNS no

I believe the delays were being caused by a combination of IPv6 and DNS, but regardless of the cause, that was the fix.