Cryptosystems: Configuring IPSec

In the next two articles, I'll demonstrate how to configure and troubleshoot an IPSec VPN on a FreeBSD system. When I first started configuring VPNs, I quickly discovered two things. First, there is more than one way to configure a VPN correctly . Along with my demonstration configuration files, I'll be including URLs to other IPSec tutorials, and you'll see for yourself that the syntax will vary slightly from tutorial to tutorial. Don't let the discrepancies bother you; instead, choose a configuration style that makes sense to you and results in a working VPN. Second, I found that I was typically left to my own devices when a VPN didn't work. There are few things in life more frustrating than following a set of instructions, only to discover that they don't work for your specific situation. Accordingly, I've included error messages which I have run across and my resolutions in the hopes that they might aid you in troubleshooting.

I'll be demonstrating a tunnel between two FreeBSD machines acting as VPN gateways. While my demonstration is specific to FreeBSD, it's possible to successfully apply the logic underlying the configurations to allow any system to access any IPSec VPN gateway. If this is your first VPN configuration, try it between two FreeBSD systems first. Once you get a handle on how a working tunnel operates and how to resolve the pitfalls you may come across, you'll be in good shape to experiment with other systems. The success of your experiments will depend upon the VPN gateway you are connecting to and how far the vendor has deviated from the IPSec standard by adding extra features.

In order for your FreeBSD system to use IPSec, you must first configure IPSec support into your kernel. If this is your first kernel build, you'll want to read through the kernel config section of the handbook first.

If you're unsure whether your kernel already has IPSec support, use this command:

sysctl -a | grep ipsec

If you don't get anything back, you need to recompile your kernel. If your tunnel is between two FreeBSD machines, both machines need IPSec support. This is how I configured my kernels. First, as the superuser, I copied the generic kernel configuration file to a file I called IPSEC:

cp /usr/src/sys/i386/conf/GENERIC /usr/src/sys/i386/conf/IPSEC

Then, using my favorite editor, I added the following three lines to the options section of /usr/src/sys/i386/conf/IPSEC:

options		IPSEC
options		IPSEC_ESP
options		IPSEC_DEBUG

While you're in the configuration file, ensure that this default line is still there; it should be unless you've removed it in a previous compile:

pseudo-device	gif

Once you've saved your changes:

cd /usr/src
make buildkernel KERNCONF=IPSEC && make installkernel KERNCONF=IPSEC

Normally I would reboot after a kernel is installed, but in this instance I'll wait until I've finished the rest of my configurations.

When you install IPSec support, you are installing the ability to use AH and ESP and to understand SADs and SPDs. However, you still have to create the policy that will be stored in that SPD and the SAs that will be stored in the SAD. It is possible to do all of this manually using a command known as setkey. But you may remember that it's better to use a key negotiation protocol to create those SAs for you on a regular basis. IKE, also known as ISAKMP, is the key negotiation protocol used by IPSec.

Currently, there are two possible ways to install IKE support on your FreeBSD system; both are found in the security section of the ports collection. The first is called racoon and the second is called isakmpd. I've found that the syntax used by racoon is easier for a novice to understand and there are more racoon resources available on the Internet, so I'll demonstrate its usage.

To install racoon:

cd /usr/ports/security/racoon
make install clean

Once this build is complete, I have the necessary ingredients for the VPN and can now concentrate on the VPN policy. That policy will be limited to the parameters supported by racoon, which can be found in man racoon.conf. I've summarized those parameters in the following table:

Remember, it is important to ensure that the policy you configure will match up on both peers. If the other VPN gateway isn't running racoon, you'll have to research that vendor's documentation to see which parameters are supported to ensure you configure the most secure policy that will result in a match on both peers.

I've decided to use the following policy:

Whenever I configure a VPN, I always write the policy parameters on a piece of paper which I can refer to as I configure the policy. Underneath the policy, I sketch out the two gateways I'll be configuring and clearly label their IP addresses:

Figure 1 -- a logical VPN diagram

You'll note that each gateway has 2 interface cards and 2 IP addresses. The external IP is the address used to connect to the Internet. The internal IP will usually be a private range address. Notice that I haven't given you my real external addresses, but have labeled them as A.A.A.A and B.B.B.B for the purposes of this article.

It only takes a few minutes to sketch out your network, but it might save you hours of troubleshooting. It is very easy to inadvertently place the wrong IP in the wrong configuration file if you don't have a sketch to refer to.

Related Reading Virtual Private Networks By Charlie Scott, Paul Wolfe, Mike Erwin

We're ready to start the configurations. I'll start with the racoon configuration file on Gateway A. The racoon port installed two sample configuration files in /usr/local/etc/racoon:

cd /usr/local/etc/racoon
ls
psk.txt.dist	racoon.conf.dist

I'll start by copying the sample file to the file which will be used by racoon:

cp racoon.conf.dist racoon.conf

Now, let's take a look at that file:

 $KAME: racoon.conf.in,v 1.18 2001/08/16 06:33:40 itojun Exp $

# "path" must be placed before it should be used.
# You can overwrite which you defined, but it should not 
# use due to confusing.
path include "/usr/local/etc/racoon" ;
#include "remote.conf" ;

# search this file for pre_shared_key with various ID key.
path pre_shared_key "/usr/local/etc/racoon/psk.txt" ;

Since I'm using a pre-shared secret as the authentication method, I'll leave the above lines as-is.

# racoon will look for certificate file in the directory,
# if the certificate/certificate request payload is received.
path certificate "/usr/local/etc/cert" ;

I removed those 3 lines as I won't be using a CA.

# "log" specifies logging level.  
#It is followed by either "notify", "debug"
# or "debug2".
#log debug;

# "padding" defines some parameter of padding.  You should not touch these.
padding
{
	maximum_length 20;	# maximum padding length.
	randomize off;		# enable randomize length.
	strict_check off;	# enable strict check.
	exclusive_tail off;	# extract last one octet.
}

As the comment indicates, the above lines should be left as-is.

# if no listen directive is specified, racoon will listen to all
# available interface addresses.
listen
{
	#isakmp ::1 [7000];
	#isakmp 202.249.11.124 [500];
	#admin [7002];		# administrative's port by kmpstat.
	#strict_address; 	# required all addresses must be bound.
}

See those last four lines with # in front of them? I kept the one with an IP address, but changed the address to the external IP of Gateway A.

# Specification of default various timer.
timer
{
	# These value can be changed per remote node.
	counter 5;		# maximum trying count to send.
	interval 20 sec;	# maximum interval to resend.
	persend 1;		# the number of packets per a send.

	# timer for waiting to complete each phase.
	phase1 30 sec;
	phase2 15 sec;
}

This section should stay as-is.

remote anonymous
{
	#exchange_mode main,aggressive;
	exchange_mode aggressive,main;
	doi ipsec_doi;
	situation identity_only;

	#my_identifier address;
	my_identifier user_fqdn "sakane@kame.net";
	peers_identifier user_fqdn "sakane@kame.net";
	#certificate_type x509 "mycert" "mypriv";

	nonce_size 16;
	lifetime time 1 min;	# sec,min,hour
	initial_contact on;
	support_mip6 on;
	proposal_check obey;	# obey, strict or claim

	proposal {
		encryption_algorithm 3des;
		hash_algorithm sha1;
		authentication_method pre_shared_key ;
		dh_group 2 ;
	}
}

remote ::1 [8000]
{
	#exchange_mode main,aggressive;
	exchange_mode aggressive,main;
	doi ipsec_doi;
	situation identity_only;

	my_identifier user_fqdn "sakane@kame.net";
	peers_identifier user_fqdn "sakane@kame.net";
	#certificate_type x509 "mycert" "mypriv";

	nonce_size 16;
	lifetime time 1 min;	# sec,min,hour

	proposal {
		encryption_algorithm 3des;
		hash_algorithm sha1;
		authentication_method pre_shared_key ;
		dh_group 2 ;
	}
}

All of that remote stuff is your Phase One policy. Notice that you can have as many remote sections as you require. If you have multiple peers using different policies, you can create a remote section for each peer by specifying the peer's IP address. Whatever you place in remote anonymous will affect all peers that don't have a specific remote section matching their IP.

sainfo anonymous
{
	pfs_group 1;
	lifetime time 30 sec;
	encryption_algorithm 3des ;
	authentication_algorithm hmac_sha1;
	compression_algorithm deflate ;
}

sainfo address 203.178.141.209 any address 203.178.141.218 any
{
	pfs_group 1;
	lifetime time 30 sec;
	encryption_algorithm des ;
	authentication_algorithm hmac_md5;
	compression_algorithm deflate ;
}

sainfo address ::1 icmp6 address ::1 icmp6
{
	pfs_group 1;
	lifetime time 60 sec;
	encryption_algorithm 3des, cast128, blowfish 448, des ;
	authentication_algorithm hmac_sha1, hmac_md5 ;
	compression_algorithm deflate ;
}

All of that sainfo stuff is the Phase Two configuration. Again, you can have sainfo sections for specified peer IPs or you can use sainfo anonymous.

Now let's see what that file looks like after I've removed the sections I don't need and inserted my own policy parameters:

 $ more /usr/local/etc/racoon/racoon.conf

# $KAME: racoon.conf.in,v 1.18 2001/08/16 06:33:40 itojun Exp $

path include "/usr/local/etc/racoon" ;

path pre_shared_key "/usr/local/etc/racoon/psk.txt" ;

# "log" specifies logging level.  It is followed by either "notify", "debug"
# or "debug2".
#log debug;

# "padding" defines some parameter of padding.  You should not touch these.
padding
{
	maximum_length 20;	# maximum padding length.
	randomize off;		# enable randomize length.
	strict_check off;	# enable strict check.
	exclusive_tail off;	# extract last one octet.
}

# if no listen directive is specified, racoon will listen to all
# available interface addresses.
listen
{
	isakmp A.A.A.A [500];
}

# Specification of default various timer.
timer
{
	# These value can be changed per remote node.
	counter 5;		# maximum trying count to send.
	interval 20 sec;	# maximum interval to resend.
	persend 1;		# the number of packets per a send.

	# timer for waiting to complete each phase.
	phase1 30 sec;
	phase2 15 sec;
}

remote B.B.B.B
{
	#exchange_mode main,aggressive;
	exchange_mode aggressive,main;
	doi ipsec_doi;
	situation identity_only;

	nonce_size 16;
	lifetime time 60 min;	# sec,min,hour
	initial_contact on;
	support_mip6 on;
	proposal_check obey;	# obey, strict or claim

	proposal {
		encryption_algorithm blowfish;
		hash_algorithm sha1;
		authentication_method pre_shared_key ;
		dh_group 5 ;
	}
}

sainfo B.B.B.B
{
	pfs_group 5;
	lifetime time 24 hour;
	encryption_algorithm blowfish ;
	authentication_algorithm hmac_sha1;
	compression_algorithm deflate ;
}

If you compare that file to my sketched out policy, you'll see that most of the configuration parameters are put into this file. Notice that I used A's external IP in the listen directive, and made Phase One and Phase Two policies for B's external IP. Since both policies need to be identical on both peers, I can simply copy that file over to Gateway B while remembering to switch the addresses so B's configuration file listens on B's address and has configuration sections for A.

Next I'll create the file that contains the pre-shared secret:

$ vi /usr/local/etc/racoon/psk.txt

#peer's external ip	pre-shared secret
B.B.B.B			dontguessme

That file needs to exist on both peers. The file on Gateway A uses B's external IP; the file on Gateway B uses A's external IP. Doublecheck for typos in the password itself, especially if you use a hard to guess password, which you should.

It's important to remember to change the default permissions of this file on both gateways or else Phase One will fail:

$ chmod 600 /usr/local/etc/racoon/psk.txt

At this point, racoon is configured and ready to go. I just need a few more configurations to tell FreeBSD about the VPN. The installation of racoon creates a startup script:

$ more /usr/local/etc/rc.d/racoon.sh

#!/bin/sh

case "$1" in
   start)
      if [ -x /usr/local/sbin/racoon ]; then
         /usr/local/sbin/racoon -f /usr/local/etc/racoon/racoon.conf \
            && echo -n ' racoon'
      fi
   ;;

   stop)
      /usr/bin/killall racoon && echo -n ' racoon'
   ;;
	
   *)
      echo "Usage: `basename $0` { start | stop }"
      exit 64
   ;;
esac

That line between if and fi starts racoon at bootup using the configuration file specified by the f switch. However, I like to modify that line so it also includes the l or logging switch which tells racoon to log to the specified file. My modified line looks like this:

/usr/local/sbin/racoon -f /usr/local/etc/racoon/racoon.conf -l 
/var/log/racoon.log && echo -n ' racoon'

Note that the output was wrapped so you could see it; when you modify that line, just do one long line without pressing enter until you are finished.

You don't have to go in and create that log file using the touch command as racoon will create the file for you.

One last thing I add to the end of that file is a command that tells the system which packets need to be sent over the tunnel. For example, on Gateway A, any packets being sent to 192.168.1.x (the encryption domain of Gateway B) need to go through the VPN tunnel. So on Gateway A, I include this line:

/sbin/route add -net 192.168.1.0/24 10.0.0.1

That line basically says, if you need to go to network 192.168.1.0, go to 10.0.0.1 so you can be routed correctly into the VPN tunnel. If you're new to CIDR prefixes, you might not recognize the /24 part. That is the other way of writing the subnet mask of 255.255.255.0 as it is really 8 bits (255) + 8 bits + 8 bits + 0 bits = 24.

On Gateway B, I do the opposite as I want to go to Gateway A's encryption domain:

/sbin/route add -net 10.0.0.0/8 192.168.1.1

The /8 is the same as the subnet mask of 255.0.0.0. Technically, that line can be added to any script that is read at bootup, so you may see it in different files as you read various tutorials. I prefer to place it in racoon.sh as it reminds me that the route is used for the VPN tunnel.

We're almost finished with the configurations. A few lines need to be added to /etc/rc.conf. I'll add these lines to Gateway A:

ipsec_enable="YES"
ipsec_file="/etc/ipsec.conf"
gif_interfaces="gif0"
ifconfig_gif0="10.0.0.1 netmask 255.0.0.0 192.168.1.1 netmask 255.255.255.0"
gifconfig_gif0="A.A.A.A netmask 255.255.240.0 B.B.B.B netmask 255.255.240.0"

The first two lines ensure IPSec support is loaded at bootup. We'll be creating that IPSec file in just a moment.

The gif_interfaces command creates a pseudo device that represents the tunnel. Technically, this isn't required in order for the tunnel to work, but it does make it a bit easier to troubleshoot the tunnel and you'll notice that most tutorials mention it. If you'd like a further explanation on this, there was a very interesting discussion in the mailing lists this past summer.

The last two lines tell your FreeBSD system which addresses to use in the headers. Remember that in tunnel mode the original IP packet contains the IP addresses of the real source and destination, which are usually private range addresses withing the two encryption domains. The new IP header created by ESP contains the external IP addresses of the two VPN gateways.

The ifconfig_gif0 command contains the internal IP addresses of the two gateways and the gifconfig_gif0 command contains the external IP addresses of the two gateways. On Gateway A, you should place Gateway A's IP addresses first, followed by Gateway B's. On Gateway B, switch the order so Gateway B's addresses are first.

The last file we need to create is ipsec.conf. At first, this file seems redundant as it creates the policy stored in the SPD. You may be thinking, "I've already configured that policy in racoon.conf". Actually, you only told racoon (or the IKE protocol) what policy parameters it should use when negotiating the SAs which will be placed in the SAD. However, racoon can't place the policy in the SPD for you; that is the job of setkey. Creating /etc/ipsec.conf means you won't have to run setkey manually every time you reboot.

My file on Gateway A looks like this:

$ more /etc/ipsec.conf

#delete all existing entries from the SAD and SPD databases
flush;
spdflush;

#add the policy to the SPD database
spdadd 10.0.0.0/8 192.168.1.0/24 any -P out ipsec
esp/tunnel/A.A.A.A-B.B.B.B/require;

spdadd 192.168.1.0/24 10.0.0.0/8 any -P in ipsec
esp/tunnel/B.B.B.B-A.A.A.A/require;

The policy lines (the two lines that start with spdadd) aren't one long wrapped line, meaning you can press enter after the word ipsec. The first policy line says whenever packets from network 10.0.0.0 need to go "out" to network 192.168.1.0, they are "required" to be encrypted by esp and to go through the tunnel that goes from Gateway A to Gateway B.

The second policy line says that whenever packets from network 192.168.1.0 need to come "in" to network 10.0.0.0, they are "required" to be encrypted by esp and to go through the tunnel that goes from Gateway B to Gateway A.

The file on Gateway B will be similar, but the addresses will be reversed so "in" and "out" match up. See why it is handy to have a sketch of your network so you can remember which address belongs where?

We've covered a lot this article. Next week, I'll show you what to watch out for when you reboot your computer after the configurations are complete. We'll then test the tunnel and work through some troubleshooting scenarios. In the meantime, you may found the following URLs useful:

• The FreeBSD Handbook's IPSec section
• http://www.kame.net/
• http://asherah.dyndys.org/~josh/ipsec-howto.txt
• Setting up a FreeBSD IPSec Tunnel

Dru Lavigne is a network and systems administrator, IT instructor, author and international speaker. She has over a decade of experience administering and teaching Netware, Microsoft, Cisco, Checkpoint, SCO, Solaris, Linux, and BSD systems. A prolific author, she pens the popular FreeBSD Basics column for O'Reilly and is author of BSD Hacks and The Best of FreeBSD Basics.

Read more FreeBSD Basics columns.

Return to the BSD DevCenter.

Copyright © 2009 O'Reilly Media, Inc.