Measuring and logging HD Radio interchannel delay

The analog FM program audio and the main digital audio are supposed to be synchronized very closely to produce a clean blend between analog and digital when necessary at the receiver. With some work, you can synchronize these down to within a few samples. The problem now, depending on your installation, is keeping it synchronized. A lot of stations are finding that their digital signal drifts in time against the analog. The drift problem is being worked on, but for now, my concern is "How do I keep aware of how well I am synchronised?"

A ham operator and FM enthusiast, Brian Beezley (K6STI), wrote a command line program to cross-correlate the delay error between the two channels of a wave file and output the results to a text file. It seems logical to me that one could automate the recording of audio from a receiver in split mode to create many samples of audio for analysis. Then, another automated script could analyze the sample audio files and output the results in a statistical format. Since it doesn't have to be done in real time, all these data points (timestamped delay error values) could be saved and graphed, showing a nice graphical view of the drift. If you use a package like OpenNMS to collect and log the data, you could even get notification when it drifted outside certain limits. That is one way to keep on top of the drift and know how much it changes.

Monitoring the network (and more) with OpenNMS

A thread on Pubtech about SNMP management prompted me to do a little searching on IT infrastructure monitoring and notification. Intermapper has been around for a while, but I didn't feel like hassling with trial serial numbers to try it out since it is commercial software. I did find a great project called OpenNMS. This is open source, free software, and it does a lot. It has been under development for several years and has been implemented in a number of large-scale IT shops. The newest versions have a ton of useful features, and there is a great web front end for managing the app. On the flip side, it was very difficult for me to install set up. Some of that I brought upon myself. I chose to install it on an Ubuntu server, which didn't have anyone to keep the opennms packages up-to-date like the RedHat folks do. I also chose to use the development version because it was said to be feature rich and stable, but I couldn't find precompiled installation packages for Ubuntu. So I downloaded the sources and compiled it myself. Since I had very little experience with that process, it took me a few tries to get all the dependencies worked out. I was ultimately successful, and it runs great. I notice that as of the 1.3.8 release there are packages available for Debian based distributions. Should make installation much easier.

The next hurdle was collecting SNMP data on some of our broadcast equipment. To add the correct SNMP OID's, I had to edit some XML files by hand, which, because of my lack of experience with the software, took a few tries to get right. I also had to edit other text files to build reports visualizing the new data that was being gathered. But, in the end, it still works great, and I can monitor data on generic network equipment as well as some special purpose hardware, like EbNo on our ContentDepot receivers, RSSI on our wireless link to the transmitter and more.

Even better, OpenNMS can send email, page, text message, even IM, if a value gets out of range. This makes it useful for getting notification if there is snow in the sat dish or interference to the wireless link or even if the transmitter shuts down. It can even be integrated into trouble ticket management software, like BestPractical RT. I really like this software and am looking forward to rolling it out for production use.

My next experiment will be to use the HTTP collection agent to pull data from our Burk ARC16 running Autopilot. I have never liked their data storage and reporting functions, so I think it would be neat to integrate the metrics like that into a system-wide monitoring app like OpenNMS. Now that I think of it, I could even turn status alarms into OpenNMS events. Back to work I go.