I've been working with IPsec for many years, mostly in tunnel mode, when building LAN-to-LAN VPN connections or for mobile worker VPNs. Recently, though, I had occastion to venture into using IPsec in transport mode, which I'd never done before. In the standard tunnel mode, entire packets are encapsulated within a single stream, encrypted, then sent on their way. This is great for LAN-to-LAN connections where you want to allow a subnet (or many subnets) on one side of the tunnel to be able to communicate with a subnet (or many) on the other side of the tunnel.

To contrast, in transport mode, IPsec merely encrypts packet payloads, leaving the IP headers intact. This fact enables easy integration into existing routing infrastructures and makes it perfect for securing host-to-host communication on untrusted networks.

What follows is a quick guide on how to set up IPsec transport mode between two linux servers. The guide is specifically written for Ubuntu 12.04 LTS, but should be easily portable to other distributions.

Before we start, let me put forth a few assumptions. We will be establishing communication between two servers - server-01 and server-02. The IP address of server-01 is 10.0.0.11 and the IP of server-02 is 10.0.0.12.

First, install prerequisite packages:

$ sudo apt-get update && sudo apt-get install openswan

IPsec core functions are provided by the kernel - "openswan" provides the user-space tools needed to easily configure connections, in addition to providing the "pluto" daemon, which takes care of the periodic re-keying that is essential to IPsec's security.

Next, add a line like this to your /etc/ipsec.secrets file.

10.0.0.11 10.0.0.12: PSK "strongpresharedkey"

Needless to say, you'll need to substitute the IP addresses of your actual servers in addition to providing a strong pre-shared key that will be used to bootstrap the IPsec connection between these two IPs.

Add this line to the bottom of /etc/ipsec.conf:

include /etc/ipsec.d/*.conf

NOTE: this line is optional - it is only required if you'd like to keep each individual IPsec connection configuration in its own file. If you don't care about this, you can just add the below IPsec config to the /etc/ipsec.conf file and be done with it.

However, if you do want to keep things clean and separate out each connection into its own file (doing this can make automated configuration very simple), then add the above line and then add the following to a new config file. Something like /etc/ipsec.d/foo-connection.conf:

conn foo-connection
  type=transport
  authby=secret
  left=10.0.0.11
  right=10.0.0.12
  pfs=yes
  auto=start

Let me digest this for you line by line.

  • conn foo-connection: arbitrary label for your connection. This can be anything you'd like.
  • type=transport: we want to use transport mode for this connection
  • authby=secret: we'll be using a pre-shared key (PSK) for this connection. This is in reference to the line we added to /etc/ipsec.secrets. IPsec can also use certificate authentication, which adds an additional level of security, but also adds complexity in configuration, so we're starting simple.
  • left=10.0.0.11: this and the next line are just denoting the IP addresses involved in this IPsec association. It does not matter which IP is "left" and which is "right".
  • right=10.0.0.12: see explanation in above bullet.
  • pfs=yes: we want to enable Perfect Forward Secrecy for this connection. In short, this drastically improves security. It's a large topic to discuss, so I'm not going to touch it now. If you want to read more, Wikipedia is your friend.
  • auto=start: We want to pro-actively initiate the IPsec association immediately. This can also be set to auto=add, in which case it waits for the other end of the connection to initiate traffic.

The above configuration needs to be duplicated on both server-01 and server-02. Their configurations can be identical with the exception of one of them should have their connection set to auto=add and one to auto=start. This way, they're not "fighting" over who should initiate the connection. In all honesty, I've not seen it cause a problem when both are set to auto=start, but it's a best practice to only have one peer initiate things.

At this point, we're ready to fire things up. On each server run this command:

$ sudo service ipsec restart

After doing this, all traffic between server-01 and server-02 should be encrypted. To verify this, run a ping between servers and then:

$ sudo tcpdump esp

This will display all ESP packets. You should see two lines (one incoming, one outgoing) for each ping. If you are not seeing ESP traffic, then something is wrong and you'll need to dig into the joy that is IPsec troubleshooting.

IPsec is a very deep subject, and there is a ton to learn. This post has only just scratched the surface. In a future post, perhaps I will cover IPsec in tunnel mode, and possibly even get into certificate authentication.

Good luck!