Startup Splunk brings search to admins

Googling may be the best way to find data on the Internet, but when it comes to searching through those impenetrable data center log files, a San Francisco startup wants you to try something new: Splunking.

Free IT resource Open Source Business Conference (OSBC) May 22-23, 2007 Sponsored by OSBC Free IT resource TechNet: More ways to know it, share it, and keep it running. Sponsored by Microsoft

Splunk Technology Inc., a 20-person company that is to be publicly unveiled at the LinuxWorld Conference and Expo this week, has developed a search engine designed to sift through the voluminous log files, message queues and even binary data generated by data center applications.

The Splunk Personal Server, which will be available for free download on Monday, can examine both live and archived files in the data center, index that information using a special algorithm, and then make the results searchable via a Web interface, said Michael Baum, the company's Chief Executive "Splunker." It is designed to make the kind of troubleshooting normally done with custom-designed scripting software much easier and more effective, he said.

"Today, when there's a problem in the data center, (system administrators) use pretty blunt stone-age instruments," he said. "It's a lot of manual labor to go through what has become a very large and fast-growing mountain of data."

Later this year, Splunk plans to release a commercial version of the Splunk Server, which will be more scalable and secure than the Personal Server. A product designed for clusters of servers will follow some time next year, he said.

The company also plans to launch a community Web site, called SplunkForge, where developers can download and develop open source "collector" software that works with the Splunk server, Baum said. "We have about a dozen projects already going on SplunkForge, where people have developed binary collectors to talk to MySQL, Oracle, or other sources of binary data," he said.

SplunkForge.org is expected to go live late Sunday, the company said.

With US$5 million in venture capital funding, raised last December, Splunk is staffed with executives from Infoseek and Yahoo Inc., where Baum previously served as vice president of e-commerce services.