Monitoring the network (and more) with OpenNMS

A thread on Pubtech about SNMP management prompted me to do a little searching on IT infrastructure monitoring and notification. Intermapper has been around for a while, but I didn't feel like hassling with trial serial numbers to try it out since it is commercial software. I did find a great project called OpenNMS. This is open source, free software, and it does a lot. It has been under development for several years and has been implemented in a number of large-scale IT shops. The newest versions have a ton of useful features, and there is a great web front end for managing the app. On the flip side, it was very difficult for me to install set up. Some of that I brought upon myself. I chose to install it on an Ubuntu server, which didn't have anyone to keep the opennms packages up-to-date like the RedHat folks do. I also chose to use the development version because it was said to be feature rich and stable, but I couldn't find precompiled installation packages for Ubuntu. So I downloaded the sources and compiled it myself. Since I had very little experience with that process, it took me a few tries to get all the dependencies worked out. I was ultimately successful, and it runs great. I notice that as of the 1.3.8 release there are packages available for Debian based distributions. Should make installation much easier.

The next hurdle was collecting SNMP data on some of our broadcast equipment. To add the correct SNMP OID's, I had to edit some XML files by hand, which, because of my lack of experience with the software, took a few tries to get right. I also had to edit other text files to build reports visualizing the new data that was being gathered. But, in the end, it still works great, and I can monitor data on generic network equipment as well as some special purpose hardware, like EbNo on our ContentDepot receivers, RSSI on our wireless link to the transmitter and more.

Even better, OpenNMS can send email, page, text message, even IM, if a value gets out of range. This makes it useful for getting notification if there is snow in the sat dish or interference to the wireless link or even if the transmitter shuts down. It can even be integrated into trouble ticket management software, like BestPractical RT. I really like this software and am looking forward to rolling it out for production use.

My next experiment will be to use the HTTP collection agent to pull data from our Burk ARC16 running Autopilot. I have never liked their data storage and reporting functions, so I think it would be neat to integrate the metrics like that into a system-wide monitoring app like OpenNMS. Now that I think of it, I could even turn status alarms into OpenNMS events. Back to work I go.

Syncing computers to Content Depot

Here are some resources on syncing computers to Content Depot with NTP.

Microsoft pages showing how to configure Windows Time service to use the Content Depot receiver as an NTP source.

Windows XP
http://support.microsoft.com/kb/314054/

Windows 2000
http://support.microsoft.com/kb/216734/

I posted previously about using the Meinberg NTP software to monitor the Content Depot devices NTP servers to indicate if they were working or not. Meinberg NTP can also be used to sync to them.

I am trying the Beagle Software ClockCard with ClockWatch Server to sync our NexGen audio server to Content Depot. So far it looks very accurate and should not drift if there are large amounts of time that we cannot synchronize. I also have some indications that NexGen provides native support for the card. I am looking into it.

Some are also using the Dimension 4 NTP client. I have not tried it, but others really like it.

Lastly, some have mentioned a program called About Time. Again, I have not tried that one, plus, it looks like older software, not updated in a while.

Tags: radio networking ntp content depot

More details on half-duplex networking with Moseley Starlink with LAN option

I wrote this in an email to a colleague who asked how I tested our network connection over our Starlink STL. I wanted to save it for future reference.
----------------------------------------------------------

Ah, I have not tried using NetCat on Win98. Thanks for the tip. I just pinged myself over UDP on my XP box. On one command prompt, I ran the program like this:

nc.exe -u -L -p 5002

On a second command prompt window, I ran the program like this:
echo "Hello World" ¦ nc.exe -u 127.0.0.1 5002

This printed out the text "Hello World" on my first command window. This should work the same between two computers on a LAN.

Now, there is one small snag that I hit with the Moseley Starlink LAN port. Forgive me if I am repeating something you are already familiar with. On a normal network (full duplex), when a host wants to open a connection to a destination IP address, it first has to turn that IP address into an Ethernet MAC address (192.168.0.1 ==> 00:01:de:74:2a:07 for instance). To do this, it uses a protocol called ARP. It sends a broadcast message out that says "Who has [dest ip]? Tell [host address]." Then, when the dest computer hears its address asked for, it sends a reply to the requesting computer. The original host who made the request then caches the address and finishes its connection. The problem for the Starlink is that a destination computer on the receive side can never send an ARP reply back to the computer that asked for it, so no hosts on the transmit side can ever figure out on their own what the MAC addresses are on the receive side, hence they can never generate any packets to send. So, the network administrator has to set up his switch or the computer that wants to send packets over the Starlink manually with the MAC information for the computers on the receive side. In Windows, for example, if you know the IP address and MAC address of a computer on the receive side of the Starlink, you can put this information into a computer on the transmit side of the Starlink with this command:

arp -s ip_address mac_address

This will add a static entry into this computer's ARP table so that it never has to ask for the address. It just knows what it is and makes the packet and away it goes. There are also commands for Cisco switches to help this situation too, but I would have to look them up.

To answer your second question, I have tested our Starlink connection. I actually used a program called VLC to encode an audio file as a stream and it sent the data using UDP to a computer on the other end running VLC as a client. I posted a link to VLC on my blog but I didn't have time then to go into detail. You might read a little about VLC and see if it something you want to try.

Tags: radio networking HD Radio