Archive for July, 2007

FreeBSD on HP BL460c Blade Servers

Tuesday, July 17th, 2007

We have standardized on HP servers and use blade servers whenever possible for our Managed client solutions.

The BL460c blades have a pair of embedded HP NC373i gigabit adapters. They have a Broadcom 5708 chipset.

Initially when we started using the C class blade servers in late 2006 we could not get networking to work under FreeBSD 6.2 RELEASE with the BL460c blade servers.

Once FreeBSD was installed it was not even seeing the “bce” adapters. We searched and searched and only found others with a similar problem.

HP doesn’t officially support FreeBSD on its gear so we could not get support there.

It also appeared that drivers had already been implemented for these adapters in an older release of FreeBSD so these “bce” adapters should work out of the box. Something done specifically with those adapters in the BL460c blade servers was causing issues.

The first step was for us to get a STABLE release of FreeBSD 6.2 several months later which apparently had further adapter driver updates as seen in CVS. We grabbed a STABLE iso image and after install the “bce” adapters were now visible and operational.

The second step was for us to configure 802.1q VLAN tagging on the NICs because of the way the blade chassis switches were uplinked to our network distribution layer. Left untagged with no VLANs the adapters wouldn’t communicate with the network.

802.1q VLAN Tagging on FreeBSD 6.2

Tuesday, July 17th, 2007

VLAN tagging allows a single network interface to access multiple networks at the same time. The multiple VLANs are presented as individual “virtual” NICs to the operating system. Each virtual NIC can have its own IP configuration.

VLAN tagging is useful when your server needs to connect to multiple networks but has a limited number of physical network adapters, or when the physical adapters are “trunked” into a single interface for speed or redundancy.

The underlying protocol is 802.1q and the server operating system, network adapter and the connected switch must support the protocol. There is other functionality within 802.1q but here we are only concerned with VLAN tagging.

The switch port(s) connected must also be configured for VLAN tagging. I will not go into switch configuration or VLAN theory and planning here.

Although I did not test it, IPv6 settings should work as well when applied to the VLANs.

————————————————————-

Manual settings

The VLAN interfaces can be turned up and configured manually at the command line.

Create the VLANs:

# ifconfig vlan_device vlan vlan_id vlandev parent_device

vlan_device is a number you assign to the interface.
vlan_id is the VLAN number to use that is configured on the connected switch.
parent_device is the physical interface name this VLAN is running on.

Example

# ifconfig vlan0 vlan 101 vlandev fxp0

Assign IP settings:

# ifconfig vlan_device ip address netmask netmask

Example

# ifconfig vlan0 10.20.30.1 netmask 255.255.255.0

An ifconfig vlan0 or an ifconfig –a will now show a vlan0 interface with the assigned IP settings.

Remove:

# ifconfig vlan0 destroy

————————————————————-

Permanent settings

The following configurations will allow the VLANs to be setup at server boot time.

1.) Edit /etc/rc.conf

The cloned_interfaces parameter will create the desired VLAN interfaces:

cloned_interfaces=”vlan20 vlan30″

Add ifconfig lines for each vlan device:

ifconfig_vlan20=”inet 192.168.0.3 netmask 255.255.255.0 vlan 20 vlandev fxp0″
ifconfig_vlan30=”inet 172.17.16.3 netmask 255.255.255.0 vlan 30 vlandev fxp0″

If the physical interface will not have its own IP address assigned, bring it up explicitly:

ifconfig_fxp0=”up”

2.) Reboot the server to verify your configurations work at boot time.

————————————————————-

Notes:

– Whether configured manually or at boot time you should see a kernel module loaded for VLAN operation.

kldstat returns a list of loaded modules. You should see if_vlan.koin the list when using VLANs.

– It is good practice to avoid using VLAN ID 1 anywhere as it is usually reserved as the Primary/Native VLAN on switches.

– Not all NICs support 802.1Q VLANs - look for VLAN_MTU and VLAN_HWTAGGING options for your NIC in ifconfig. man vlan provides further info. I found though that my adapter (bce0) was not listed in man vlan but it did support the necessary options and worked with VLAN tagging.

————————————————————-

Older versions of FreeBSD

Older versions of FreeBSD may not load the kernel module automatically. In these cases you will need to modify your kernel to enable the VLAN tagging functionality.

To compile this driver into the kernel, place the following lines in your kernel configuration file:

device miibus
device vlan

- or add something like the following line to add the vlan devices directly into the kernel

pseudo-device vlan 2 # IEEE 802.1Q VLAN Support

Alternatively, to load the driver as a module at boot time, place the following line in loader.conf (/boot/loader.conf):

if_vlan_load=”YES”

Note: I did not test these kernel settings on older versions of FreeBSD myself. This was gleaned from other sources on the Internet.