opus:interactive | If These Servers Could Talk

The Goal:

We needed a way to protect a set of load balanced web servers that host public facing web sites for a client. The web servers each run a copy of the hosting control panel DirectAdmin on top of FreeBSD 6.2 Release.

The Problem:

DirectAdmin licensing requires a real public IP address on the server’s external interface to protect against using the license multiple times by hiding it on a private network. Many different flavors of control panel software have this requirement.

This, of course, eliminates the ability to take the easy way out and NAT the servers behind a firewall.

We would have simply purchased a firewall appliance for this installation (we implement GB-2000 firewall appliances by GTA Inc. however an additional requirement was to use existing 1U server hardware and not purchase an additional appliance.

Solution:

The end solution was to install FreeBSD 6.2 Release on the 1U server hardware and utilize packet filter (pf) as a transparent bridge to meet the IP addressing requirements.

Hardware load balancing was used to load balance HTTP traffic to the web servers but will not be discussed here.

Howto:

1.) Install FreeBSD 6.2.

We have a checklist list of tasks to perform to install and lock down our production servers. Follow your own best practices to get a basic install of FreeBSD 6.2 running and patched. Install the minimal amount of options and packages necessary.

You will need, or at least you will most likely want, a third NIC installed in the server. In a transparent bridge the WAN and LAN interfaces become “transparent” and no longer take an IP address. So without the third NIC installed and connected to your network you will have no way to remotely manage the server. A benefit of this though is that without an IP address to attack your transparent bridging firewall itself would be free from attack.

Pf is available in a default install by re-compiling the kernel with specific changes made, or by enabling pf via kernel loadable module.

We re-compiled the kernel. The options below were added at the end of the kernel source and the new kernel compiled:

# pf support
device pf # Packet Filter firewall
device pflog # PF logging facility
device pfsync # PF state syncing

# ALTQ support
options ALTQ
options ALTQ_CBQ
options ALTQ_RED
options ALTQ_RIO
options ALTQ_HFSC
options ALTQ_PRIQ

2.) Configure your third NIC with an IP address and verify you can remotely access your server.

In the /etc/rc.conf file we have the following definition for the management IF:

ifconfig_fxp0=”inet 123.123.123.2 netmask 255.255.255.0″

We will be building pf rules for this NIC as well to protect the firewall itself.

3.) Create the bridge between the two desired interfaces.

Use your favorite editor to edit /etc/rc.conf and enable the bridge

Add:

cloned_interfaces=”bridge0″
ifconfig_bridge0=”addm bge0 addm nfe0 up”
ifconfig_bge0=”up”
ifconfig_nfe0=”up”

In this case we are bridging the two interfaces bge0 and nfe0.

Use your favorite editor to edit /etc/sysctl.conf

Add:

net.link.bridge.pfil_onlyip=1
net.link.bridge.pfil_member=1
net.link.bridge.pfil_bridge=0

4.) Enable the use of pf on your server.

Use your favorite editor to edit /etc/rc.conf and enable the use of pf

Add:

pf_enable=”YES” # enable PF (load module if required)
pf_rules=”/etc/pf-bridge.conf” # rules definition file for pf
pf_flags=”" # additional flags for pfctl startup
pflog_enable=”YES” # start pflogd(8)
pflog_logfile=”/var/log/pflog” # where pflogd should store the logfile
pflog_flags=”" # additional flags for pflogd startup

5.) Build the firewall ruleset.

First make a copy of the default ruleset and designate it as a bridging ruleset.

# cp /etc/pf.conf /etc/pf-bridge.conf

Use your favorite editor to edit /etc/pf-bridge.conf. Place your ruleset within the pf-bridge.conf file and save the changes.

Here is the sample ruleset we used: pf-bridge_generic.txt

6.) Apply the rules and enable the firewall.

Finally to actually enable a new ruleset we need to tell pf to read the config file. This would also automatically happen upon reboot.

# pfctl –f /etc/pf-bridge.conf
# pfctl –e

That’s it! You will now need to go through and test the bridge and verify you can access what you intended to allow access to, and that what you wanted to block is now blocked. Hopefully you still have access to the management interface as well. The best test will be to perform some form of vulnerability testing against IPs behind your firewall and the firewall itself.

Some notes on the ruleset specifically:

There is really nothing in the ruleset that designates the firewall as a transparent bridge other than the absence of NAT rules. The bridge built in the OS itself in the /etc/rc.conf file is where the bridging is applied.

The IP addresses in the variable and table definitions will obviously have to be updated to fit a different environment.

Many options exist for pf and there are full books dedicated to the art of pf rulesets and using pf in general. This ruleset for example could be expanded to make more use of AltQ for QoS and added protection against DoS attacks.

Hurdles:

The first major hurdle we ran into had to do with Multiple MAC address tables on the switch (or lack thereof). We wanted to use a single switch to handle the connectivity for both the inside (LAN) and outside (WAN) of my transparent bridge. To do this we created two VLANs on the switch so that the WAN IF of the bridge connected to VLAN 1 and the LAN IF and the web servers connected to VLAN 2.

The problem is that because of the bridging, the MAC addresses of the web server network adapters were now appearing in both VLANs on the switch thoroughly confusing it.

It turns out that the switch we were using – an older HP Procurve – only had a single MAC address table.

The interim solution was to use two switches, one for each side of the transparent bridge. A better solution will be to use a switch that has more than one MAC address table so that we can use only a single switch for this solution.

The second hurdle is really only the fact that it can take time to perfect the pf ruleset. There are so many options and more than one way to do the same thing. Keep working till it works out correctly.

Some additional commands for managing pf:

# pfctl -s rules : list current parsed rules
# pfctl -f filename : reload the ruleset with the specified file
# pfctl -d : disable pf
# pfctl -e : enable pf
# pfctl -R /etc/pf.conf : enable rules from specified file
# pfctl -s rules -v : hit stats for each rule

View current log with TCPDump:

Log specific tcp packets to a different log file with a large snaplen
(useful with a log-all rule to dump complete sessions)

# pflogd -s 1600 -f suspicious.log port 80 and host evilhost

Display binary logs:
# tcpdump -n -e -ttt -v -r /var/log/pflog

Display the logs in real time (this does not interfere with the operationof pflogd):

# tcpdump -nexttti pflog0