Adding Nagios Logs to Splunk

Now that we have all these systems working correctly under OSX its time to start making them work together a little.

First up I want to add the nagios logs to splunk. This is very easy, you can get this off the splunk site here. But I’ll recap exactly what I did for my setup here.

1. Click Manager in the upper right-hand corner of Splunk Web.
2. Under System configurations, click Data Inputs.
3. Click Files and directories.
4. Click New to add an input
I choose Monitor a file or directory
6. Specify the path to the file:
With my setup it is:
7. Under Host Heading
I choose constant value
8. Under Source Type
I choose Automatic
9. Click Save

Thats it now your nagios logs show up in splunk. Pretty easy stuff.

Custom Weather Notifications with Growl

Last night I download Prowl on my iPhone and setup my growl to work with it. It’s very cool stuff together, i’ve been using growl forever.

Anyway tonight I was reading in this thread in the prowl forum where one poster is using growl notifications for weather. Not just any weather but really local weather. Now if you live in or around a big town, most weather apps are pretty accurate for you area. But when you live out in the sticks like I do, they are only close most of the time.

Anyway tonight I set up this excellent pair of perl scripts as outlined here from IBM: Develop your own weahter maps and alerts. Which is a very cool script that will allow you pinpoint your location. I used Photoshop to create the base map from the layers. Once followed all the instructions, some things are not exactly clear at first, but if your familiar with perl reading the code sorts it all out. I setup the notify scripts to send the messages to growl via the growlnotify command.

Now once that was all setup I created a simply bash script that would delete the old Radar overlay, pull the current Radar overlay needed and run the perl weather scripts. I then stuck that script in my crontab. So if I’m at my computer I get notified and if i’m away from my computer i get a push notification to my phone. Very cool stuff. Of course I could just look outside to see if it is raining 🙂

Splunk on OSX

Another tool that I like use is Splunk. Now we use a different set of tools for log monitoring and management at work, but I enjoy using splunk at home.

The good thing about Splunk on OSX is that they provide you with a .dmg to download and .pkg to install. Takes longer to download than to install. Once the install is done just start it up and log in.


Next up for reinstall is OSSEC. OSSEC is an Open Source Host-basted Intrusion Detection System. I also had this installed before the i reinstalled OSX.

To install OSSEC just follow the default instructions and everything works out just fine. Note, you’ll have to start this manual after each reboot, I’m sure there is a way to add it to autostart, but I haven’t gotten there yet.

To install the OSSEC-WUI follow the instructions up to the point before running the script, it will not work on OSX (client anyway, not sure about server). All you need to do to get it working is first change the permission on the whole folder and files to _www. Then you need to add the _www user to the ossec group. That is done with the following command:
sudo dscl . -append /Groups/ossec GroupMembership _www

Thats it now its up and running and you have a nice interface for it.

Cacti on OSX

To continue on with monitoring my home network environment with the some of the tools I use to monitor my work environment I’m reinstalling Cacti. Now before the reload of my mac I had cacti running and graphs for at least 3 years. Now while I don’t mind losing that historical data i do mind loosing all the custom scripts that I had written to monitor so of the now SNMP devices on my network, I will probably pull that drive out and copy the data over soon. Note to self include all these config files in my future backup plan.

My first run at installing Cacti was via Macports, which I’ve never tried before. What I discovered is that, the version on macports wouldn’t install with the plugins support. So I did the way I normally do an installed from source. No special notes for osx here, it just works. Same goes for adding the plugin support, worked out of the box, following there install instructions.

So all I can recommend is following the install instructions and install from ports and you’ll be in business. The only thing that I think doesn’t work is the Localhost memory usage. But I’ll be digging into that soon and getting it sorted out with the mac version.

VIM and Nagios

I edit just about all my nagios files at the command line. I have found a nagios.vim file that highlights the syntax and really helps when working with the files.
First you can get it here: nagios syntax
Next simply follow the install details listed. I made the following changes to fit my nagios installation on osx via macports:
on the line that starts au BufNew, i changed the line to:
au BufNewFile,BufRead /opt/local/etc/nagios/objects/*.cfg set filetype=nagios

Another note, just in case you haven’t already done so you can auto enable the syntax for vim buy putting the following line in your .vimrc
:syntax enable

More Nagios Plugins on OSX

More about my nagios tweaking on OSX.

Adding check_dig (checks the DNS Server)
First there is no command defintion for this already setup. You’ll need to edit: /opt/local/etc/nagios/objects/commands.cfg
and add the following:
# ‘check_dig’ command definition
define command{
command_name check_dig
command_line $USER1$/check_dig $ARG1$

in your router,localhost, or where ever this services is add:
define service{
use generic-service
host_name linksys-wrt54g
service_description DNS
check_command check_dig!-H $HOSTADDRESS$ -l -A “+tcp”

Next up is check_dhcp
Nothing much here expect on my OSX it doesn’t get a mac so you’ll have to include that in your command: for example mine looks like:
check_dhcp!–mac=00:14:51:62:57:b3 -s $HOSTADDRESS$
(note I made that mac up, not sure exactly if thats a good idea or bad idea)

Next up is check_disk
while check_local_disk is already setup to monitor your disk, I wanted to added my 2 Firewire drives, myservices definition looks like this for the first one.

define service{
use local-service
host_name localhost
service_description FireWire1
check_command check_local_disk!10%!5%!/Volumes/Firewire1

Next up check_ftp
Nothing special here, works out of the box.

Next up check_http
Works out of the box

next up check_ifoperstatus and check_ifstatus
works out of the box

next up check_ide_smart
Was not compiled and installed from Macports on my system.
After doing some research the files need to compile this are linux only and after some searching there is no osx version of it.

next up check_mysql
This was not compiled and installed from Macports on my system for some reason, I think i didn’t have mysql installed at the time. Not sure if it would have worked then. Any way what I ended up having todo was downloading the source file of nagios-plugins. And running the configure like so:
./configure –with-mysql=/opt/local/lib/mysql5

then I was able to copy the check_mysql and check_mysql_query to /opt/local/libexec/nagios

And they worked fine.

Works out of the box, here is an example of the file locations on my mac:
./check_nagios -e 5 -F /opt/local/var/nagios/status.dat -C /opt/local/bin/nagios

Thats about it for all the stuff that I have and the standard plugins.

Nagios on OSX : Re-schedule the next check of this service

An out of the port install of Nagios from macports doesn’t allow you to re-schedule the next check of this service and a few other things. The problem is that the files have the permissions for nagios but not the www user, which is what your useing from the browser. But this is an easy fix.
$sudo dscl . -append /Groups/nagios GroupMembership www
$sudo dscl . -append /Groups/nagios GroupMembership _www

Getting working under osx

I often work and tweak things to get them going on OSX (not server) and then forget how I did it. Then I’ll end up wiping the machine and have to everything over and spending a ton of time on it. I should have documented getting nagios going properly and ndoutils and nagvis, but i didn’t.

But here is what I did to get the plugin working on OSX. At least part of it any way I crashed and lost some of the notes. check_ndo does exactly what its it checks to make sure that ndoutils is writing to the mysql database as expected and if not alerts you.

One weird thing that it took me lots of poking around with was why the plugin wouldn’t work from the command line. The first problem was that it was using the default perl install and not the perl install from Macports. Also it was looking in the default for linux location for the file not the default for OSX nagios via macports.

Here is how I did it.
First edit the file. $sudo vi /opt/local/libexec/nagios/
change the use lib line to:
use lib “/opt/local/libexec/nagios”;

Next to make things easier i create a command in the commands.cfg
$sudo vi /opt/local/etc/nagios/objects/commands.cfg
I add the following:
# check_ndo
define command{
command_name check_ndo
command_line /opt/local/bin/perl /opt/local/libexec/nagios/ -H $HOSTNAME$ -P 3306 -d $ARG1$ -u $ARG2$ -p $ARG3$ -i $ARG4$ -t 300

Next I added it to my localhost config, but it can be added anywhere:
$sudo vi /opt/local/etc/nagios/objects/localhost.cfg
# Service to check that ndoutils is working
define service{
use generic-service
host_name localhost
service_description NDOUTILS
check_command check_ndo!dbname!dbuser!dbpassword!default!300

Hopefully this will help me remember this in the future and anyone else who tries it.