Can a federation tackle the data management puzzle?

I could probably fill up my column just reporting on who's buying whom -- or who's partnering with whom -- in storage, but I would probably end up just talking about big names such as EMC and their latest acquisitions.

Free IT resource Virtualization Insights from Top Experts - Learn how virtualization gets real! Sponsored by Dell Free IT resource TechNet: More ways to know it, share it, and keep it running. Sponsored by Microsoft

(Incidentally, that hypothetical column would probably focus on EMC's latest carefully selected pick: ProActivity Software Solutions, a provider of BPM applications. The incorporation of ProActivity likely will open another interesting phase in EMC's strategy for world domination, perhaps adding to the "need to know my data" cognitive layer another actionable layer: "How does this affect my business processes?")

Regardless of what EMC does with ProActivity, analyzing and classifying gigantic volumes of data remains one of any company’s more formidable challenges. Obviously, data indexing and search are prerequisites to analysis and classification, and handling those processes efficiently is a major requirement.

"Anybody can scale to handle billions of documents, but it's going to take a ton of hardware to do it," states Robert Lancaster, vice president of channel development at Fast Search and Transfer (FAST). "We can handle between 30 and 50 million documents on a single, commodity dual-CPU server."

FAST is an enterprise software vendor with lots of experience in data retrieval and indexing. "We license our search infrastructure technology to third-party software and, increasingly, hardware vendors," Lancaster explains.

FAST's flagship application is the Enterprise Search Platform, which the company has licensed to an impressive list of OEMs, including well-known names such as Documentum -- now part of EMC -- and Lexis-Nexis, according to Lancaster.

Could storage vendors use the same technology? You bet. In fact, my conversation with FAST was designed to inform me about the company’s latest announcement, Instream for Data Classification, which is "a specific solution for archiving and storage vendors," Lancaster explains.

Among some of the early adopters are EMC, in products such as Centera SEEK and ChargeBack Reporting, and Avamar Axion search and de-duping features, Lancaster says. He adds that there are numerous areas for application of the Instream technology, including legal and electronic discovery and regulation compliance.

Isn't the technology from FAST also a perfect complement to archiving solutions? Of course it is, and the proof resides in the final -- but no less important -- partner name Lancaster mentions: Archivas.

Archivas’ inclusion in the list is not unusual these days because the company, sensing a renewed interest for this once-snubbed market segment, has been weaving a fabric of partnerships to complement its flexible data-archiving solution.

Adding to the buzz, Archivas just released Arc 1.8, a new version of its clustered archiving solution that adds -- among other features -- integrated search and discovery (with a little help from FAST) and improves scalability to a stupefying capacity. That scalability would be as much as 2.5PB and more than 2 billion user files on a single cluster, says Jeff Spotts, vice president of marketing at Archivas.

At the same time, Archivas announced three new partnerships in HSM (hierarchical storage management). The new partnerships involve products such as CaminoSoft Managed Server, also sold as part of CA BrighStor HSM; Enigma Data PARS (Project Archiving and Retrieval System), a popular application in the oil and gas industry; and Scentric Destiny, offering a combined archiving solution for files, e-mail, and databases.

"These are joint marketing and sales arrangements," Spotts explains. "We have done the technical work to integrate and test our respective products, which gives us the framework to approach the market jointly."

That sounds reassuring, but can a group of partners effectively compete with the likes of EMC, HDS, and others in the archiving space and its adjacent territories? 

I don't see why not -- after all, open-systems computing was born from a similar contrast between all-encompassing, single-provider mainframe solutions and a federation of servers, OSes, and applications from a variety of vendors.

Join me on The Storage Network blog with questions or comments.