CLUMEQ transforms rundown particle accelerator into high-efficiency cooling enclosure
HPC consortium discovers circular shape of concrete structure yields significant cooling efficiencies
The Université Laval in Quebec, Canada, had two problems. First, its campus was home to a run-down particle accelerator, constructed in the 1960s, that needed to be decommissioned. Second, the university and 11 of its fellow institutions, members of an HPC consortium, needed a place to construct a state-of-the-art supercomputer. With a little ingenuity -- and a devotion to embracing sustainable practices -- the group was able to transform the 36-foot-wide, 65-foot-high circular concrete silo into an effective cooling enclosure for its supercomputer.
Transforming the silo into a home for a new data center presented some unusual challenges for CLUMEQ (Consortium Laval, Université du Québec, McGill and Eastern Quebec). The final design concept comprised a topology where three levels of server racks are arranged along a circle, creating an inner hot-air cylindrical core and an outer ring-shaped cold-air plenum. The large floor cross-section of the cold-side plenum results in very low air velocity, almost no turbulence (thanks to the absence of corners), and thus uniform temperature and pressure, according to Marc Parizeau, professor at Université Laval and deputy director of CLUMEQ.
"Having a single annular-shaped cold aisle with a large cross-section and thus very low air velocity is probably close to ideal if one wants to air cool today's high-power density racks without using rear-door heat exchangers or other technologies that require bringing water near the servers," Parizeau said.
The main cooling system, located in the basement, pulls the hot air down from the center using energy-efficient variable drive fans. The hot air is cooled by forcing it through highly efficient, custom-designed walls of coils connected to the campus chilled water loop. Designers considered employing liquid cooling, but "simulations demonstrated that [benefits would be] marginal compared with our 120,000 CFM blowing capacity, and not worth the risk -- and costs -- of putting water above the servers."
Heat waste generated by the supercomputer is put to good reuse. During eight months of the year, it's transferred from the chilled water return to the campus hot-water loop to provide heating for the school, thus reducing energy bills.
The supercomputer itself was built by Sun using the company's Constellation blade system. It's composed of 7,680 Intel Nehalem cores with 24TB of RAM and 1 petabyte of high-performance, high-availability parallel storage. The server hardware itself is "quite energy efficient, but not significantly more than the competition," Parizeau said.
Beyond enjoying the benefits of using an existing structure to house its supercomputer, the Université Laval and CLUMEQ estimate the silo design results in annual savings of more than 1.5 million kWh, compared with a traditional data center. Transforming the silo into a data center likely costs more than going the conventional square-build, raised-floor route, Parizeau said, "but this does not take into account the higher efficiency of the silo design, nor the fact that we recycled a building that was almost impossible to reuse for anything else. It may have cost a little more, but we got more for the money -- and there were no budget overruns."