Nortel proving out virtual mash-ups on the Grid

In previous blog entries, we've discussed the common conundrum faced by organizations with large, distributed data sets ... do you move the data to the compute, or do you move the compute to the data? There's been a fair bit of discussion about GridFTP and related protocols for bulk file transfers for Grid environments. And, of course, the very nature of Grid computing is to provide resource pools that can be dynamically provisioned to jobs in the queue.

But a very exciting and NEW, related discussion is taking place around "mashups" of virtual machines and virtual network resources. This new research area is being driven by Dr. Franco Travostino -- who heads Grid activities in the capacity of Director for Advanced Technology and Research at Nortel and is co-Director for the Infrastructure Area at the GGF -- and who is without question one of the most interesting players on the networking side of the enterprise Grid discussion.

Travostino and his team have made some fascinating breakthroughs with Xen hypervisors and Nortel's own DRAC (PDF: 'Dynamic Resource Allocation Controller').

"In Seattle, at Supercomputer 2006, we created a demo by which we took some Xen-based virtual machines that were crunching some particular, computation-intensive tasks ... and we moved them to Amsterdam," said Travostino. "And then from Amsterdam, we moved them to Chicago, and them from Chicago back to Seattle. And in spite of the tens of thousands of miles, the impact on the applications was less than one second. So that's pretty mind-boggling, to think about having a fully featured Linux environment running lots of applications, and teleporting all that across the world with such minimal disruption. DRAC is the "network middleware" that makes this long-haul migration possible at the network level. Specifically, DRAC puts in place a short-lived deterministic network service, on demand. As well, it preserves the sessions with any remote client."

Later this year, Travostino is running an essay chronicling the experiment in the Elsevier Future Generation Computer Systems publication ... but to paraphrase a few of the specific reasons he sites for why virtual network resources + virtual machines is so exciting to the Grid community:

* "[Often] it is impossible to bring data (or devices) closer to the computation engines. This may result from policy limitations (e.g., a data set that is embargoed from export) or capacity limitations (e.g., a data set is exceedingly large, thus adding an unwieldy preamble to the computation phase). An application's style of operation may exacerbate the latter characteristic (e.g., an application that infers from intermediate computation results which data sets are required next, lending itself to traveling salesman optimizations);

* It is desirable to load balance computation workloads with a scope that transcends the confines of individual data centers. One policy may dictate that during off hours the workload be consolidated into fewer data centers over a regional area to limit operating expenses or power consumption. Another policy might dictate that the workload "tracks the moon" to harvest spare computational power in different geographical areas that happen to be off business hours;