opus:interactive | If These Servers Could Talk

This write-up assumes a working copy of FreeBSD 7.0.  It was built using 7.0-RELEASE.  It should work on FreeBSD 6.x-STABLE and future versions of FreeBSD 7.  The package versions listed were current as of this writing but may have been updated by the time someone uses this howto.

This is a basic setup of Smokeping.  There are many extra features that I do not touch on here such as multi-target graphs, alerting, slaves, agents and additional probe types.  Check the online documentation for further info.

Please let me know if you run into typos or other technical issues when implementing this.

1.) First let’s update the ports collection.

Setup the update:

Make it look something like this:

Run the update:

Update installed ports:

2.) Install the necessary packages.

Descriptions of packages and uses here: http://oss.oetiker.ch/smokeping/doc/smokeping_install.en.html

Installed by default with FreeBSD install

Installed by default with FreeBSD install

Socket6

Net:DNS

3.) Configure the packages.

Add:

(within the <IfModule alias_module> section)

Add:

(within the <IfModule dir_module> section)

Add smokeping.cgi after index.html on the DirectoryIndex line to allow smokeping.cgi to load as a default document.

It should look like:

Allow the startup of Apache:

Add the following to /etc/rc.conf

Start Apache:

Edit the variables in the following files appropriately

Variable example settings in the first section:

*** Alerts ***

The “*** Targets ***” section is where you define your Smokeping targets and build the navigation menu.  It is a little too in depth to cover here.  Play around with it to figure out how it works.

In depth configuration info is here: http://oss.oetiker.ch/smokeping/doc/smokeping_config.en.html

There are some configuration samples here: http://oss.oetiker.ch/smokeping/doc/smokeping_examples.en.html

Verify the path to Speedy:

Customizations to the Smokeping web page templates can be made in the following config files:

Set file system permissions to make the img folder and files writeable:

Allow the startup of Smokeping:

Add the following to /etc/rc.conf

Start Smokeping:

Operational Notes:

 index.html at /usr/local/www/apache22/data/

This redirects traffic sent to the root of your server to a different domain since this traffic obviously doesn’t need to be hitting the Smokeping server.

#/usr/local/etc/rc.d/smokeping restart


This write-up makes the following assumptions:

• Working copy of FreeBSD 6.2.
• Build: 6.2-RELEASE.

Which should work on 6.2-STABLE and 7.0 as well.

Please let me know if you run into typos or other technical issues when implementing this.

1. Download the latest binaries from mysql.com.

At write-up time this was:

mysql-5.0.45-freebsd6.0-i386.tar.gz
mysql-standard-4.1.22-unknown-freebsd6.0-i386.tar.gz
2. Install MySQL 4.

Uncompress the binary source.

# cd /usr/local
# gunzip < /path/to/mysql-VERSION-OS.tar.gz | tar xvf -
# ln -s full-path-to-mysql-VERSION-OS mysql4

Create the daemon user and group.

# pw groupadd mysql4
# pw useradd -n mysql4 -c "" -g mysql4 -d /nonexistent -s /usr/sbin/nologin

Set the file permissions.

# cd /usr/local/mysql4
# chown -R root:mysql4 .

Run the setup database setup script.

# scripts/mysql_install_db --user=mysql4

Copy the startup script to the proper location.

# cp /usr/local/mysql4/support-files/mysql.server /usr/local/etc/rc.d/mysql4.server.sh

Change two variables in the start script.

# pico /usr/local/etc/rc.d/mysql4.server.sh

Change “basedir=” to “basedir=/usr/local/mysql4″
Change “datadir=/usr/local/mysql/data” to “datadir=/usr/local/mysql4/data”
Change “pid_file=” to “pid_file=/var/run/mysql4/mysql4.pid”

Create configuration files.

Copy one of the sample configuration files based on your server usage replacing xxxx in the line below.

# cp /usr/local/mysql4/support-files/my-xxxx.cnf /usr/local/mysql4/data/my.cnf

Change variables in the configuration file.

# pico /usr/local/mysql4/data/my.cnf

Add a variable at the top of the [mysqld] section - “user = mysql5″
Change the “port = 3306″ variables to “port = 3304″
Change the “socket = /tmp/mysql.sock” variables to “socket = /tmp/mysql4.sock”

Create a run directory for the MySQL 4 process and set permissions on it.

# mkdir /var/run/mysql4
# chown -R mysql4:mysql4 /var/run/mysql4
3. Install MySQL 5.

Uncompress the binary source.

# cd /usr/local
# gunzip < /path/to/mysql-VERSION-OS.tar.gz | tar xvf -
# ln -s full-path-to-mysql-VERSION-OS mysql5

Create the daemon user and group.

# pw groupadd mysql5
# pw useradd -n mysql5 -c "" -g mysql5 -d /nonexistent -s /usr/sbin/nologin

Set the file permissions.

# cd /usr/local/mysql5
# chown -R root:mysql5 .

Run the setup database setup script.

# scripts/mysql_install_db --user=mysql5

Copy the startup script to the proper location.

# cp /usr/local/mysql5/support-files/mysql.server /usr/local/etc/rc.d/mysql5.server.sh

Change a few variables in the start script.

# pico /usr/local/etc/rc.d/mysql5.server.sh

Change “basedir=” to “basedir=/usr/local/mysql5″
Change “datadir=” to “datadir=/usr/local/mysql5/data”
Change “pid_file=” to “pid_file=/var/run/mysql5/mysql5.pid”
Change “server_pid_file=” to “server_pid_file=/var/run/mysql5/mysql5.pid”
Change “user=mysql” to “user=mysql5″

Create configuration files.

Copy one of the sample configuration files based on your server usage replacing xxxx in the line below.

# cp /usr/local/mysql5/support-files/my-xxxx.cnf /usr/local/mysql5/my.cnf

Change variables in the configuration file.

# pico /usr/local/mysql5/my.cnf

Add a variable at the top of the [mysqld] section - “user = mysql5″
Change the “port = 3306″ variables to “port = 3305″
Change the “socket = /tmp/mysql.sock” variables to “socket = /tmp/mysql5.sock”

Create a run directory for the MySQL 5 process and set permissions on it.

# mkdir /var/run/mysql5
# chown -R mysql5:mysql5 /var/run/mysql5
4. Start the daemons.
# /usr/local/etc/rc.d/mysql4.server.sh start
# /usr/local/etc/rc.d/mysql5.server.sh start
5. Post installation configuration.

MySQL4:

Connect to MySQL

# /usr/local/mysql4/bin/mysql -u root -P3304 -S/tmp/mysql4.sock

Remove the anonymous account.

mysql> DELET FROM mysql.user WHERE Host='localhost' AND User='';

Set the root password

mysql> SET PASSWORD FOR 'root'@'localhost' = PASSWORD('difficultpassword1');
mysql> SET PASSWORD FOR 'root'@'server.domain.com' = PASSWORD('difficultpassword1');

Create a backupoperator account for backup script access to all databases.

mysql> GRANT ALL PRIVILEGES ON *.* TO 'backupoperator'@'localhost' IDENTIFIED BY 'difficultpassword2';

Apply changes.

mysql> FLUSH PRIVILEGES;

Quit.

mysql> QUIT;

MySQL5:

Connect to MySQL

# /usr/local/mysql5/bin/mysql -u root -P3305 -S/tmp/mysql5.sock

Remove the anonymous account.

mysql> DROP USER '';

Set the root password.

mysql> SET PASSWORD FOR 'root'@'localhost' = PASSWORD('difficultpassword1');
mysql> SET PASSWORD FOR 'root'@'server.domain.com' = PASSWORD('difficultpassword1');
mysql> SET PASSWORD FOR 'root'@'127.0.0.1' = PASSWORD('difficultpassword1');

Create a backupoperator account for backup script access to all databases

mysql> GRANT ALL PRIVILEGES ON *.* TO 'backupoperator'@'localhost' IDENTIFIED BY 'difficultpassword2';

Quit

mysql> QUIT;

That’s it! Create your mysql users and databases and connect your applications.

Notes:

1. Command line connections to mysql now require additional connection info to specify port and socket:
   MySQL4 - # /usr/local/mysql4/bin/mysql -u username -ppassword -P3304 -S/tmp/mysql4.sock
MySQL5 - # /usr/local/mysql5/bin/mysql -u username -ppassword -P3305 -S/tmp/mysql5.sock
2. Remote connections to mysql will require specifying the correct port in the connection string instead of assuming port 3306.

I recently attended a Microsoft Certified Systems Engineer Boot Camp, put on by TECHPROS Group, in an attempt to obtain my MCSE. Yet, the certification wasn’t the only goal, I intended to meet all the criteria for the certification - in one week.

Back in August, I hopped on a plane from Portland, OR to Irvine, CA. To start the trip off, the plane had maintenance issues, so as we started to take off - literally as the front of the plane left the ground - the pilot immediately landed the plane. The plane barely stopped in time before its tires rolled off the end of the runway. For the next two hours we were stalled, waiting for the mechanical issues to be resolved.

MCSE credentials are the certifications for professionals who analyze the business requirements and design and implement the infrastructure for business solutions based on the Microsoft Windows Server System.

Because of this travel hurdle, I arrived three hours late to the boot camp. Missing certification orientation and dinner might phase one’s moral, but here at opus:interactive, it takes more than a false start at the beginning of the race to thwart a victory - a lot more.

The boot camp was held at a rented house in the hills outside Irvine, CA. It was a very nice house with plenty of space and rooms for all fifteen of the boot camp attendees (below). TECHPROS had great content setup for us in terms of lectures and labs (below). The first three days we worked through labs with VMware as well as lectures. I finished the first lab Sunday night before turning in around 1AM. The second day I finished the second lab by dinner time and was ready to start studying for the tests. I crammed for 14 hours straight and attempted my first test on Tuesday:

I passed with a perfect score 1000/1000! The rest of the day was spent going through more labs and some simulations TECHPRO had setup to prepare us for the simulations on 70-291. I studied until around 2AM and was ready to take more tests on Wednesday.

I was the only student to attempt three tests in one day and the ones I chose were:

I ended up passing all three that day with a 984/1000 on the XP test! Even with these three tests behind me, I didn’t stop. I continued cramming and working on other labs and lectures for the rest of the day into the wee hours of Thursday, turning in around 3AM.

I wanted to continue the testing insanity and scheduled three more tests for Thursday as follows:

I was again the only student to attempt three tests that passed them all in one day.

The camp was winding down and there was only one more testing opportunity before the flight back to Portland on Friday afternoon. I schedule one final test on Friday morning which would achieve my MCSE +Messaging.

I ended up taking a second test because I had two hours to kill before catching a ride to the airport. I took the following tests:

Both of these test I completed and passed, which brought my total of successfully passed tests to nine in four days! (Phew, what a relief!) I was literally brain dead at this point.

One week later I felt the itch again and decided I should take the last elective for the Security add-on to the MCSE. I quickly scheduled this test:

Another perfect score of 1000/1000! Completing that exam gave me my MCSE +Security.

Why stop there! I thought and continued a study for the new Hosting Specialization Exam:

This test encapsulates what we offer here at opus:interactive and as such: I nailed it with a 987/1000 score! That achieved me the MCTS: Hosting Specialization certification as well.

In recent months we’ve been internally studying the ITIL Foundations and was finally ready to take that exam. This particular exam was administered by EXIN the Examination Institute for Information Science and is the first step in us implementing ITIL for our organization. I studied all the material I could find and ran some practice exam engines. I was finally ready to take the test. I aced that test with a perfect score of 1000/1000! Obtaining my ITIL Foundations Certification.

A lot of testing and a lot of proof of just how excellent our organization and team are. Some might even say: the best in the business.

With the passage of time, there is always technological challenges that crop up for existing clients, which is why as a managed service provider we step in to eagerly correct issues so our clients need not (after all, this is what they are paying us for.) These challenges manifest themselves either through user growth, hardware limitations or in worse case scenarios: hardware failures. Recently our monitors noticed that a five year old Windows 2000 Server was about to start digging a little virtual six-foot-deep grave and this is where we stepped into the challenge.

I quickly stabilized their old server allowing it to keep running while we worked on a migration to a new solution. While this was happening, Jeremy (our beef-jerky-loving sales representative) sold the client one of our new dedicated managed HP c-Class BladeSystem servers. The web server will be migrated to Windows 2003 and the SQL server migrated to SQL 2005 on Windows 2003 as well. Additionally, I replaced their old Cisco PIX 501 firewall.

After the migration of content, I began setting things up, including the VPN tunnels to the third party development company and the integration with the billing system. Once this was completed, I began full regression Quality Assurance.

After testing, it was quickly discovered that the integration with the billing engine was not working. The reason was the rather old code (Active Server Pages, ASP, version 3.0) which had been written by a third-party development company back in 1999. The code they produced relied on a component called URLFetch, which didn’t work inside of the new operating system environment.

Here is the particular code call:

This component is a COM Wrapper for a few lines of Java code that allow you to, basically, screen scrape sites and return them for use in the code. Since the billing system is on an internal network behind firewalls and a VPN, it can’t be served up direct to the end-user, so it has to be done from the code on the web server. Since this was an old component it was pretty much assumed that it wouldn’t work on Windows Server 2003 and IIS 6.

The research began about this component and the specific error that was being thrown by the code, which was:

I quickly found this Microsoft Knowledge base (KB) article and grabbed filemon from sysinternals to check it out.

Using depends.exe (from Dependency Walker) I looked into finding the missing .dll that I figured was trying to be loaded behind the scenes. This was confirmed and the missing dependent file was DWMAPI.dll.

From there a little more research uncovered another team having difficulty with the same situation. From this information the .dll was found and downloaded.

However, this new .dll didn’t produce results and I had to start again, from ground-zero.

It was at this point that I began to deduct that this old component would not work on the Windows 2003 and IIS6. We then contacted the client and gave them the harrowing news that the possibility that they may need to re-write their code existed. Even in the face of this possibility, we wanted to assure that all avenues had been explored and I continued researching. Next was to review the component itself and how it was written - to define what dependencies it required. This additional digging uncovered the following information:

This was the final ‘a-ha!’ moment. I then quickly downloaded the Microsoft Virtual Machine and voilà, it worked! (The new SUN Java SDK was already tried before this and it didn’t work.)

The end result: The client’s code now works great, on the new environment, with no need to update the code-base and I got to celebrate the moment with a Dr. Pepper.

The Goal:

We needed a way to protect a set of load balanced web servers that host public facing web sites for a client. The web servers each run a copy of the hosting control panel DirectAdmin on top of FreeBSD 6.2 Release.

The Problem:

DirectAdmin licensing requires a real public IP address on the server’s external interface to protect against using the license multiple times by hiding it on a private network. Many different flavors of control panel software have this requirement.

This, of course, eliminates the ability to take the easy way out and NAT the servers behind a firewall.

We would have simply purchased a firewall appliance for this installation (we implement GB-2000 firewall appliances by GTA Inc. however an additional requirement was to use existing 1U server hardware and not purchase an additional appliance.

Solution:

The end solution was to install FreeBSD 6.2 Release on the 1U server hardware and utilize packet filter (pf) as a transparent bridge to meet the IP addressing requirements.

Hardware load balancing was used to load balance HTTP traffic to the web servers but will not be discussed here.

Howto:

1.) Install FreeBSD 6.2.

We have a checklist list of tasks to perform to install and lock down our production servers. Follow your own best practices to get a basic install of FreeBSD 6.2 running and patched. Install the minimal amount of options and packages necessary.

You will need, or at least you will most likely want, a third NIC installed in the server. In a transparent bridge the WAN and LAN interfaces become “transparent” and no longer take an IP address. So without the third NIC installed and connected to your network you will have no way to remotely manage the server. A benefit of this though is that without an IP address to attack your transparent bridging firewall itself would be free from attack.

Pf is available in a default install by re-compiling the kernel with specific changes made, or by enabling pf via kernel loadable module.

We re-compiled the kernel. The options below were added at the end of the kernel source and the new kernel compiled:

# pf support
device pf # Packet Filter firewall
device pflog # PF logging facility
device pfsync # PF state syncing

# ALTQ support
options ALTQ
options ALTQ_CBQ
options ALTQ_RED
options ALTQ_RIO
options ALTQ_HFSC
options ALTQ_PRIQ

2.) Configure your third NIC with an IP address and verify you can remotely access your server.

In the /etc/rc.conf file we have the following definition for the management IF:

ifconfig_fxp0=”inet 123.123.123.2 netmask 255.255.255.0″

We will be building pf rules for this NIC as well to protect the firewall itself.

3.) Create the bridge between the two desired interfaces.

Use your favorite editor to edit /etc/rc.conf and enable the bridge

Add:

cloned_interfaces=”bridge0″
ifconfig_bridge0=”addm bge0 addm nfe0 up”
ifconfig_bge0=”up”
ifconfig_nfe0=”up”

In this case we are bridging the two interfaces bge0 and nfe0.

Use your favorite editor to edit /etc/sysctl.conf

Add:

net.link.bridge.pfil_onlyip=1
net.link.bridge.pfil_member=1
net.link.bridge.pfil_bridge=0

4.) Enable the use of pf on your server.

Use your favorite editor to edit /etc/rc.conf and enable the use of pf

Add:

pf_enable=”YES” # enable PF (load module if required)
pf_rules=”/etc/pf-bridge.conf” # rules definition file for pf
pf_flags=”" # additional flags for pfctl startup
pflog_enable=”YES” # start pflogd(8)
pflog_logfile=”/var/log/pflog” # where pflogd should store the logfile
pflog_flags=”" # additional flags for pflogd startup

5.) Build the firewall ruleset.

First make a copy of the default ruleset and designate it as a bridging ruleset.

# cp /etc/pf.conf /etc/pf-bridge.conf

Use your favorite editor to edit /etc/pf-bridge.conf. Place your ruleset within the pf-bridge.conf file and save the changes.

Here is the sample ruleset we used: pf-bridge_generic.txt

6.) Apply the rules and enable the firewall.

Finally to actually enable a new ruleset we need to tell pf to read the config file. This would also automatically happen upon reboot.

# pfctl –f /etc/pf-bridge.conf
# pfctl –e

That’s it! You will now need to go through and test the bridge and verify you can access what you intended to allow access to, and that what you wanted to block is now blocked. Hopefully you still have access to the management interface as well. The best test will be to perform some form of vulnerability testing against IPs behind your firewall and the firewall itself.

Some notes on the ruleset specifically:

Hurdles:

The first major hurdle we ran into had to do with Multiple MAC address tables on the switch (or lack thereof). We wanted to use a single switch to handle the connectivity for both the inside (LAN) and outside (WAN) of my transparent bridge. To do this we created two VLANs on the switch so that the WAN IF of the bridge connected to VLAN 1 and the LAN IF and the web servers connected to VLAN 2.

The problem is that because of the bridging, the MAC addresses of the web server network adapters were now appearing in both VLANs on the switch thoroughly confusing it.

It turns out that the switch we were using – an older HP Procurve – only had a single MAC address table.

The interim solution was to use two switches, one for each side of the transparent bridge. A better solution will be to use a switch that has more than one MAC address table so that we can use only a single switch for this solution.

The second hurdle is really only the fact that it can take time to perfect the pf ruleset. There are so many options and more than one way to do the same thing. Keep working till it works out correctly.

Some additional commands for managing pf:

# pfctl -s rules : list current parsed rules
# pfctl -f filename : reload the ruleset with the specified file
# pfctl -d : disable pf
# pfctl -e : enable pf
# pfctl -R /etc/pf.conf : enable rules from specified file
# pfctl -s rules -v : hit stats for each rule

View current log with TCPDump:

Log specific tcp packets to a different log file with a large snaplen
(useful with a log-all rule to dump complete sessions)

# pflogd -s 1600 -f suspicious.log port 80 and host evilhost

Display binary logs:
# tcpdump -n -e -ttt -v -r /var/log/pflog

Display the logs in real time (this does not interfere with the operationof pflogd):

# tcpdump -nexttti pflog0