If you haven't already seen Cacti, take a look now. I'll wait...
Back already? Great.
For what seems like eons, MRTG has been the graphing/trending tool du jour for throughput data. Tons of hacks have turned MRTG into a graphing tool for other data, such as temperature probes, disk utilization, mail statistics, ad infinitum. Tobi Oetiker's RRDTool has been around nearly as long, providing a solid round-robin database backend to store the data for just about anything that fits as a counter, and absolute, or what have you. Cacti is an extensive PHP framework around RRDTool, providing a Web GUI to add/delete/manage monitored devices and present graphs in a very elegant fashion.
I'd played with Cacti years ago when it was in it's infancy. Now, it's all grown up. The newest version supports the most common data gathering tools, and has facilities for custom additions, however Byzantine the structure may be. If you want to graph router/firewall ifInOctet/ifOutOctet data, that's simple. Graphing CPU utilization from Net-SNMP hosts is equally simple, and built into the base distribution. Adding custom data is a bit of a challenge, as it requires intimate knowledge of the inner workings of Cacti, and a bit of a brain-bend on how exactly the Cacti magic happens. Make no mistake, you will need to make several attempts at custom code unless it's simple SNMP queries. The usage of non-SNMP data for per-instance queries is decidedly non-obvious.
I found that I had a need to graph FLEXlm license utilization across multiple servers in multiple locations, with multiple sub-applications that needed individual graphs. Inputting all this data by hand would have taken eons, so being lazy, I wrote a series of Cacti XML templates and perl scripts to do all this work for me.
Basically, parsing FLEXlm license daemon output is rather nasty. It's a textual adventure and slow. So I wrote a poller daemon in perl that populates a shared hash using Tie::ShareLite. The poller runs every 4.5 minutes, while the accompanying query script runs during the Cacti polling run, referencing that shared hash. The result is that gathering data on the license utilization of hundreds of individual applications is instantaneous to Cacti, and only requires a single query to each lmgrd process on a license server from the poller.
Also, I wound up writing a query script and XML template to track datacenter temperatures via Sensatronics TempTrax probes with Cacti.
In any event, check this project out. It's better than many commercial packages, and with RRDTool 1.2, the visuals are stunning. For the first time in forever, there aren't any MRTG scripts in my crontab. Amazing.