opus:interactive | If These Servers Could Talk

We have standardized on HP servers and use blade servers whenever possible for our Managed client solutions.

The BL460c blades have a pair of embedded HP NC373i gigabit adapters. They have a Broadcom 5708 chipset.

Initially when we started using the C class blade servers in late 2006 we could not get networking to work under FreeBSD 6.2 RELEASE with the BL460c blade servers.

Once FreeBSD was installed it was not even seeing the “bce” adapters. We searched and searched and only found others with a similar problem.

HP doesn’t officially support FreeBSD on its gear so we could not get support there.

It also appeared that drivers had already been implemented for these adapters in an older release of FreeBSD so these “bce” adapters should work out of the box. Something done specifically with those adapters in the BL460c blade servers was causing issues.

The first step was for us to get a STABLE release of FreeBSD 6.2 several months later which apparently had further adapter driver updates as seen in CVS. We grabbed a STABLE iso image and after install the “bce” adapters were now visible and operational.

The second step was for us to configure 802.1q VLAN tagging on the NICs because of the way the blade chassis switches were uplinked to our network distribution layer. Left untagged with no VLANs the adapters wouldn’t communicate with the network.

VLAN tagging allows a single network interface to access multiple networks at the same time. The multiple VLANs are presented as individual “virtual” NICs to the operating system. Each virtual NIC can have its own IP configuration.

VLAN tagging is useful when your server needs to connect to multiple networks but has a limited number of physical network adapters, or when the physical adapters are “trunked” into a single interface for speed or redundancy.

The underlying protocol is 802.1q and the server operating system, network adapter and the connected switch must support the protocol. There is other functionality within 802.1q but here we are only concerned with VLAN tagging.

The switch port(s) connected must also be configured for VLAN tagging. I will not go into switch configuration or VLAN theory and planning here.

Although I did not test it, IPv6 settings should work as well when applied to the VLANs.

————————————————————-

Manual settings

The VLAN interfaces can be turned up and configured manually at the command line.

Create the VLANs:

# ifconfig vlan_device vlan vlan_id vlandev parent_device

vlan_device is a number you assign to the interface.
vlan_id is the VLAN number to use that is configured on the connected switch.
parent_device is the physical interface name this VLAN is running on.

Example

# ifconfig vlan0 vlan 101 vlandev fxp0

Assign IP settings:

# ifconfig vlan_device ip address netmask netmask

Example

# ifconfig vlan0 10.20.30.1 netmask 255.255.255.0

An ifconfig vlan0 or an ifconfig –a will now show a vlan0 interface with the assigned IP settings.

Remove:

# ifconfig vlan0 destroy

————————————————————-

Permanent settings

The following configurations will allow the VLANs to be setup at server boot time.

1.) Edit /etc/rc.conf

The cloned_interfaces parameter will create the desired VLAN interfaces:

cloned_interfaces=”vlan20 vlan30″

Add ifconfig lines for each vlan device:

ifconfig_vlan20=”inet 192.168.0.3 netmask 255.255.255.0 vlan 20 vlandev fxp0″
ifconfig_vlan30=”inet 172.17.16.3 netmask 255.255.255.0 vlan 30 vlandev fxp0″

If the physical interface will not have its own IP address assigned, bring it up explicitly:

ifconfig_fxp0=”up”

2.) Reboot the server to verify your configurations work at boot time.

————————————————————-

Notes:

– Whether configured manually or at boot time you should see a kernel module loaded for VLAN operation.

kldstat returns a list of loaded modules. You should see if_vlan.koin the list when using VLANs.

– It is good practice to avoid using VLAN ID 1 anywhere as it is usually reserved as the Primary/Native VLAN on switches.

– Not all NICs support 802.1Q VLANs - look for VLAN_MTU and VLAN_HWTAGGING options for your NIC in ifconfig. man vlan provides further info. I found though that my adapter (bce0) was not listed in man vlan but it did support the necessary options and worked with VLAN tagging.

————————————————————-

Older versions of FreeBSD

Older versions of FreeBSD may not load the kernel module automatically. In these cases you will need to modify your kernel to enable the VLAN tagging functionality.

To compile this driver into the kernel, place the following lines in your kernel configuration file:

device miibus
device vlan

- or add something like the following line to add the vlan devices directly into the kernel

pseudo-device vlan 2 # IEEE 802.1Q VLAN Support

Alternatively, to load the driver as a module at boot time, place the following line in loader.conf (/boot/loader.conf):

if_vlan_load=”YES”

Note: I did not test these kernel settings on older versions of FreeBSD myself. This was gleaned from other sources on the Internet.

This script will change the ownership of user objects in the target database to the specified user.

This situation often occurs when importing a database from another SQL server where a user account existed that does not, and will not exist on the new SQL server.

Run this script to update user object ownership before clearing the old/incorrect user from the database.

This script could be turned into a stored procedure as well and accept the new username as a passed parameter.

There are multiple ways to do this some of which are probably more glamorous – but hey, it works.

——————————————————————————

/*
This script will change the ownership of user objects in the target database to the specified user.
This situation often occurs when importing a database from another SQL server where a user account existed that does not, and will not exist on the new SQL server.
Run this script to update user object ownership before clearing the old/incorrect user from the database.
*/

SET NOCOUNT ON

/* Set the new owner name here in place of the ‘dbo’ user */
DECLARE @New_Owner_Name Sysname
SET @New_Owner_Name = ‘dbo’

DECLARE @Object_ID Int, @Object_Name Sysname, @Owner_ID Int, @Owner_Name Sysname, @Owner_Object_Name Sysname
DECLARE CUR_ReplaceObjectOwner CURSOR FOR

/* Run this select query by itself to see the set of user objects you are working with */
SELECT so.id AS “Object ID”, so.name AS “Object Name”, so.uid AS “Owner ID”, su.name AS “Owner Name”, ‘[' + su.name + '].’ + ‘[' + so.name + ']‘ AS “Owner + Object Name”
FROM sysobjects so, sysusers su
WHERE so.uid = su.uid
AND so.xtype IN (’U',’P',’FN’,'V’)
AND so.name NOT LIKE ‘%dt_%’
AND so.name NOT IN (’syssegments’, ’sysconstraints’)
ORDER BY xtype

/* Get the results from the query and load the first row into the cursor */
OPEN CUR_ReplaceObjectOwner
FETCH NEXT FROM CUR_ReplaceObjectOwner
INTO @Object_ID, @Object_Name, @Owner_ID, @Owner_Name, @Owner_Object_Name

WHILE @@FETCH_STATUS = 0
BEGIN
IF @New_Owner_Name != @Owner_Name
/* Make the object owner change here */
EXEC sp_changeobjectowner @Owner_Object_Name, @New_Owner_Name

/* Load the next row into the cursor */
FETCH NEXT FROM CUR_ReplaceObjectOwner
INTO @Object_ID, @Object_Name, @Owner_ID, @Owner_Name, @Owner_Object_Name
END

/* Clean up */
CLOSE CUR_ReplaceObjectOwner
DEALLOCATE CUR_ReplaceObjectOwner
SET NOCOUNT OFF

GO

——————————————————————————

The Goal:

We needed a way to protect a set of load balanced web servers that host public facing web sites for a client. The web servers each run a copy of the hosting control panel DirectAdmin on top of FreeBSD 6.2 Release.

The Problem:

DirectAdmin licensing requires a real public IP address on the server’s external interface to protect against using the license multiple times by hiding it on a private network. Many different flavors of control panel software have this requirement.

This, of course, eliminates the ability to take the easy way out and NAT the servers behind a firewall.

We would have simply purchased a firewall appliance for this installation (we implement GB-2000 firewall appliances by GTA Inc. however an additional requirement was to use existing 1U server hardware and not purchase an additional appliance.

Solution:

The end solution was to install FreeBSD 6.2 Release on the 1U server hardware and utilize packet filter (pf) as a transparent bridge to meet the IP addressing requirements.

Hardware load balancing was used to load balance HTTP traffic to the web servers but will not be discussed here.

Howto:

1.) Install FreeBSD 6.2.

We have a checklist list of tasks to perform to install and lock down our production servers. Follow your own best practices to get a basic install of FreeBSD 6.2 running and patched. Install the minimal amount of options and packages necessary.

You will need, or at least you will most likely want, a third NIC installed in the server. In a transparent bridge the WAN and LAN interfaces become “transparent” and no longer take an IP address. So without the third NIC installed and connected to your network you will have no way to remotely manage the server. A benefit of this though is that without an IP address to attack your transparent bridging firewall itself would be free from attack.

Pf is available in a default install by re-compiling the kernel with specific changes made, or by enabling pf via kernel loadable module.

We re-compiled the kernel. The options below were added at the end of the kernel source and the new kernel compiled:

# pf support
device pf # Packet Filter firewall
device pflog # PF logging facility
device pfsync # PF state syncing

# ALTQ support
options ALTQ
options ALTQ_CBQ
options ALTQ_RED
options ALTQ_RIO
options ALTQ_HFSC
options ALTQ_PRIQ

2.) Configure your third NIC with an IP address and verify you can remotely access your server.

In the /etc/rc.conf file we have the following definition for the management IF:

ifconfig_fxp0=”inet 123.123.123.2 netmask 255.255.255.0″

We will be building pf rules for this NIC as well to protect the firewall itself.

3.) Create the bridge between the two desired interfaces.

Use your favorite editor to edit /etc/rc.conf and enable the bridge

Add:

cloned_interfaces=”bridge0″
ifconfig_bridge0=”addm bge0 addm nfe0 up”
ifconfig_bge0=”up”
ifconfig_nfe0=”up”

In this case we are bridging the two interfaces bge0 and nfe0.

Use your favorite editor to edit /etc/sysctl.conf

Add:

net.link.bridge.pfil_onlyip=1
net.link.bridge.pfil_member=1
net.link.bridge.pfil_bridge=0

4.) Enable the use of pf on your server.

Use your favorite editor to edit /etc/rc.conf and enable the use of pf

Add:

pf_enable=”YES” # enable PF (load module if required)
pf_rules=”/etc/pf-bridge.conf” # rules definition file for pf
pf_flags=”" # additional flags for pfctl startup
pflog_enable=”YES” # start pflogd(8)
pflog_logfile=”/var/log/pflog” # where pflogd should store the logfile
pflog_flags=”" # additional flags for pflogd startup

5.) Build the firewall ruleset.

First make a copy of the default ruleset and designate it as a bridging ruleset.

# cp /etc/pf.conf /etc/pf-bridge.conf

Use your favorite editor to edit /etc/pf-bridge.conf. Place your ruleset within the pf-bridge.conf file and save the changes.

Here is the sample ruleset we used: pf-bridge_generic.txt

6.) Apply the rules and enable the firewall.

Finally to actually enable a new ruleset we need to tell pf to read the config file. This would also automatically happen upon reboot.

# pfctl –f /etc/pf-bridge.conf
# pfctl –e

That’s it! You will now need to go through and test the bridge and verify you can access what you intended to allow access to, and that what you wanted to block is now blocked. Hopefully you still have access to the management interface as well. The best test will be to perform some form of vulnerability testing against IPs behind your firewall and the firewall itself.

Some notes on the ruleset specifically:

Hurdles:

The first major hurdle we ran into had to do with Multiple MAC address tables on the switch (or lack thereof). We wanted to use a single switch to handle the connectivity for both the inside (LAN) and outside (WAN) of my transparent bridge. To do this we created two VLANs on the switch so that the WAN IF of the bridge connected to VLAN 1 and the LAN IF and the web servers connected to VLAN 2.

The problem is that because of the bridging, the MAC addresses of the web server network adapters were now appearing in both VLANs on the switch thoroughly confusing it.

It turns out that the switch we were using – an older HP Procurve – only had a single MAC address table.

The interim solution was to use two switches, one for each side of the transparent bridge. A better solution will be to use a switch that has more than one MAC address table so that we can use only a single switch for this solution.

The second hurdle is really only the fact that it can take time to perfect the pf ruleset. There are so many options and more than one way to do the same thing. Keep working till it works out correctly.

Some additional commands for managing pf:

# pfctl -s rules : list current parsed rules
# pfctl -f filename : reload the ruleset with the specified file
# pfctl -d : disable pf
# pfctl -e : enable pf
# pfctl -R /etc/pf.conf : enable rules from specified file
# pfctl -s rules -v : hit stats for each rule

View current log with TCPDump:

Log specific tcp packets to a different log file with a large snaplen
(useful with a log-all rule to dump complete sessions)

# pflogd -s 1600 -f suspicious.log port 80 and host evilhost

Display binary logs:
# tcpdump -n -e -ttt -v -r /var/log/pflog

Display the logs in real time (this does not interfere with the operationof pflogd):

# tcpdump -nexttti pflog0