For example, Merlin Securities built a high-performance cluster out of standard, relatively inexpensive hardware, but the application had to be developed in-house from the ground up.
Clusters and Grids
The basic premise of HPC is simple. Instead of running compute-intensive applications on one large, specialized system, high-performance clusters and grids divide up the processing load among anywhere from two to thousands of separate servers, workstations, or even PCs. The actual architecture used, however, will vary depending on the nature of the application and where the hardware resides.
Forrester Research divides clustering and grid-computing architectures into three categories: uncoupled, loosely coupled, and tightly coupled. The uncoupled architecture, best exemplified by Web server load balancing, is more relevant to handling streams of small requests than for HPC applications.
In the loosely coupled architecture, a workload scheduler, usually running on a head server, splits up large application requests into many smaller, parallel, independent tasks and distributes them, along with small amounts of data, among the servers making up the cluster. The job management software may or may not have to aggregate the results.
For this scenario to work, the partitioned tasks must have high compute-to-data ratios and no interdependencies or order-of-execution requirements. One good example is a query search against a huge database in which the query is run concurrently against many separate database fragments. According to Forrester, this method is appropriate for such tasks as mathematically intensive risk calculations, engineering design automation and simulation, life sciences, pharmaceutical tasks such as protein folding, and animated film rendering.
Hewitt Associates, a global human resources outsourcing and consulting firm, uses a loosely coupled cluster to process what-if scenarios for its defined benefit (pension) plans on its customer Web site. These calculations can be numerically intensive, depending on the customer’s assumptions and the number of plan renegotiations, mergers, and acquisitions that occurred during an employee’s term of employment. With help from IBM and grid software vendor DataSynapse , Hewitt was able to split off the most intense calculations to its cluster of Intel-based blade servers, now approaching 40 in number.
Although many installations of this type consist of a single dedicated departmental or datacenter server cluster, another way to implement low-cost HPC is to distribute work across a number of shared, geographically dispersed resources in what is known as a grid. A grid can run across a few company departments or datacenters, or it can cross company boundaries to partner sites and service providers. For example, Nationwide Financial takes advantage of a concept called cycle harvesting, in which desktop PCs and workgroup servers are activated for grid computing when they are idle.
Today, however, dedicated clusters are by far the most common scenario. “When I go out and talk to people, I see lots of dedicated clusters running a single application, only a handful of shared grids spanning multiple geographies, and no examples of grids spanning multiple firms,” Forrester’s Gillett says.