ONLamp.com    
 Published on ONLamp.com (http://www.onlamp.com/)
 See this if you're having trouble printing code examples


Virtualization with FreeBSD Jails

by Dan Langille
03/09/2006

This article shows how I created a jail under FreeBSD 5. Though FreeBSD 6.0 has come out since I wrote this article, the strategy should remain the same. I'll update the article with any changes should anything be different.

I have written previously about jails on FreeBSD 4. The goal of this jail is the creation of a test environment for a project I've been working on. Until recently, I've been providing a dedicated machine for exclusive use by the Bacula network backup project. That system ran regression tests on FreeBSD. In a recent consolidation of hardware, I replaced several older machines with one newer machine. I wanted to dispose of the computer used by the Bacula project and move them to a more powerful computer. However, I didn't want them to have exclusive use of this system. I wanted to use the same computer and not have us interfere with each other.

Lift and Separate

Jails can separate different processes and keep them apart so they cannot interfere with each other. For example, you could run Apache in a jail and keep it away from everything else on the machine. Should someone find an exploit in Apache and use it to compromise your system, the intruders can only do what the jail allows them to do. A jail can consist of a full operating system, or a single executable.

The solution I used was to create a virtual machine for use by the Bacula project. I had recently acquired a Pentium 4 2.4GHz machine. It was pretty fast, so I decided to use this for system for my own development purposes. It will also be sitting idle for long periods of time, so I might as well let some else use it, as well. I don't want them to have access to the things I'm working on, so I decided to put them in a jail.

From within a jail, they are chrooted and cannot see anything outside of the jail. At the same time, it appears to them as if they are running on their own machine with their own operating system. As far as they know, they have their own computer and nobody else is on the system.

Running a virtual system within a jail is a good solution if you want to provide someone with resources, but don't want them to have complete control over your system. A jail can help you deal with issues of security and access, and improve the usage of existing resources, all at the same time.

Jail Documentation

The main document for creating a jail is man jail. I followed the instructions listed under "Setting up a Jail Directory Tree." I used those instructions to create the jail. You will need the full source tree for the system you going to create. I used the /usr/src/ directory I had from my most recent build world.

There's one step from man jail that I did not follow. I left Sendmail (actually, Postfix) running. I just changed it so that it did not listen on all IP addresses. I added this setting to /usr/local/etc/postfix/main.cf:

inet_interfaces = $myhostname

That allows the jail to run its own mail server.

I put my jail at /home/jail/192.168.0.155.bacula. This is the value I assigned to D in the instructions. After you have installed the jail, if you peek inside that directory, you'll see it looks just like the root directory of a typical FreeBSD system:

[dan@dfc:/home/jail/192.168.0.155.bacula] $ ls
COPYRIGHT       etc       libexec        root           usr
bin             home      mnt            sbin           var
boot            kernel    proc           sys
dev             lib       rescue         tmp
[dan@dfc:/home/jail/192.168.0.155.bacula] $

Terminology: Host Versus Jail

The host environment is the main system and is where you first install FreeBSD on the computer. It is in the host environment that you create a jail. The Bacula project will do their testing in the jail. They have access to the jail and only the jail. They will not have access to the host environment at all.

This concept of host environment and jail environment will come up again in this article. It is important that you understand what each one is.

In this example, the host environment is at IP address 192.168.0.100 and the jail is at 192.168.0.155.

Modifying Other Daemons

Most daemons will listen to whatever IP addresses are available to them. After starting your jail, if you try to ssh to it, you will not get into it. You'll be in the host environment instead. To get into the jail environment via ssh, you need to:

Host Environment syslogd

This entry in /etc/rc.conf tells syslogd to not listen on any IP address.

syslogd_flags="-ss"

That allows syslogd to run in both the host and the jail environments.

Host Environment inetd

This entry in /etc/rc.conf tells inetd to listen on a specific IP address. This address is that of the host environment:

inetd_flags="-wW -C 60 -a 192.168.0.100"

Note that the first part of the flags in that line is from /etc/defaults/rc.conf:

inetd_flags="-wW -C 60" # Optional flags to inetd

Host Environment sshd

To alter the host environment sshd so it listens only to host environment IP addresses, modify /etc/ssh/sshd_config and set the IP address for the Listen directive:

ListenAddress 192.168.0.100

Then restart the main sshd process:

kill -HUP `cat /var/run/sshd.pid`

Use telnet to verify that the host environment is not listening on the jail address:

$ telnet 192.168.0.155 22
Trying 192.168.0.155...
telnet: connect to address 192.168.0.155: Connection refused
telnet: Unable to connect to remote host

If you don't get a connection, the host environment is not listening. This assumes that you have not yet started sshd in the jail environment.

Jail Environment sshd

To start sshd in the jail environment, add the following line to /etc/rc.conf:

sshd_enable="YES"

Jail Environment syslogd

In addition, I also swapped console output to /var/log/messages, as shown in this snippet from /etc/syslogd.conf:

#*.err;kern.warning;auth.notice;mail.crit    /dev/console
*.err;kern.warning;auth.notice;mail.crit     /var/log/messages
Mastering FreeBSD and OpenBSD Security

Related Reading

Mastering FreeBSD and OpenBSD Security
By Yanek Korff, Paco Hope, Bruce Potter

Configuring the Jail

Next, read the part of the man page titled "Configuring the Jail." It will tell you how to configure a few settings within the jail. I made these changes to the jail directly from the host environment (that is, I did not start the jail; I modified the files from outside). These are things I changed, but I can't point to a man page as to why it's a good thing to do these things:

Starting the Jail for the First Time

From man jail, to start a jail, issue the command:

[root@mtwenty:/home/dan] # jail /usr/jails/192.168.0.155 bacula.example.org 192.168.0.155\
    /bin/sh
#

That prompt (#) indicates you are now in the jail environment. Now you can run the start up processes:

# sh /etc/rc
Loading configuration files.
bacula.example.org
Setting hostname: bacula.example.org.
Generating nsswitch.conf.
Generating host.conf.
Starting syslogd.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout
Starting local daemons:.
Updating motd.
/etc/rc: WARNING: Setting entropy source to blocking mode.
====================================================
Type a full screenful of random junk to unblock
it and remember to finish with Enter. This will
timeout in 300 seconds, but waiting for
the timeout without typing junk may make the
entropy source deliver predictable output.

Just hit Enter for fast+insecure startup.
====================================================
kern.random.sys.seeded: 1
jalkjlkajdflkajdfl iur opiquv dropuivwaopieuaoijdfl;uiop9^[[12~84718e0
r9invpfinadfpisad;ifsda;lsajdfl lk;kasf;kladfs
Generating public/private rsa1 key pair.
Your identification has been saved in /etc/ssh/ssh_host_key.
Your public key has been saved in /etc/ssh/ssh_host_key.pub.
The key fingerprint is:
5c:48:47:4f:e0:c5:a2:ed:71:bc:83:b5:42:3f:95:e4 root@bacula.example.org
Generating public/private dsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_dsa_key.
Your public key has been saved in
/etc/ssh/ssh_host_dsa_key.pub.
The key fingerprint is:
4d:bb:af:fa:b1:4b:43:cc:47:b6:78:44:ad:4e:ef:1f root@bacula.example.org
Generating public/private rsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_rsa_key.
Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub.
The key fingerprint is:
9b:b5:26:98:f8:0d:da:bb:2c:57:75:d1:c4:58:52:c1 root@bacula.example.org
Starting sshd.
Starting cron.
Local package initialization:.

Sun Sep 11 17:22:42 EDT 2005
#

For the most part, this looks exactly like a normal startup.

I had some problems with ps:

# ps auwx
ps: bad namelist

This usually indicate a kernel that is not in sync with world. To fix this problem, I repeated some of the steps under man 8 jail:

# make distribution DESTDIR=$D
# mount_devfs devfs $D/dev
# cd $D
# ln -sf dev/null kernel

In hindsight, I think I missed the mount_devfs step. Symptoms included getting logged in by ssh, but then the screen would freeze. Performing these steps fixed that problem.

Starting and Stopping the Jail Automagically

I found an interesting tool for starting and stopping a jail: sysutils/jailutils. I installed it in the host environment.

Using this tool, I created this start/stop script:

#!/bin/sh

case "$1" in
   start)
       mount_devfs devfs /usr/jails/192.168.0.155/dev     && \
       mount -t procfs proc /usr/jails/192.168.0.155/proc && \
       /usr/local/sbin/jstart /usr/jails/192.168.0.155 bacula.example.org 192.168.0.155 \
          /bin/sh /etc/rc > /dev/null && echo -n ' jail bacula.example.org'
       ;;
   stop)
       /usr/local/sbin/jkill bacula.example.org > /dev/null && echo -n ' jail' && \
       umount /usr/jails/192.168.0.155/proc && \
       umount /usr/jails/192.168.0.155/dev
       ;;
   *)
       echo "Usage: `basename $0 {start|stop}" >&2
       ;;
esac

exit 0

This is a very limited script. It doesn't check that a jail is already running before starting it. That would be a nice addition. If you want to add it, I look forward to your patch.

In addition, you might want to add this to the host environment's /etc/sysctl.conf:

security.jail.set_hostname_allowed

Under 4.*, this variable had a slightly different name.

Jails Run Well

Jails run virtual machines very well. They look very much like real systems. You must look pretty close to be able to tell you're in a jail. My jail allows the Bacula developers to have a machine of their own. It also allows me to keep their work totally separate from my own.

You can use a jail used to deal with security issues and to increase the utilization of an existing machine while giving everyone their own virtual machine. There's no reason why you couldn't run many different jails on the same computer.

Dan Langille runs a consulting group in Ottawa, Canada, and lives in a house ruled by felines.


Return to the BSD DevCenter.

Copyright © 2009 O'Reilly Media, Inc.