Friendster scales the network with open source

“We worked to customize MySQL. We had great support from [MySQL AB, the parent company of the software]. We wouldn’t be able to do this with Oracle,” Lunt says. “The licensing costs would kill us. ... It’s a good product, but not that much better to justify the cost.”

Free IT resource Open Source Business Conference (OSBC) May 22-23, 2007 Sponsored by OSBC Free IT resource Virtualization Insights from Top Experts - Learn how virtualization gets real! Sponsored by Dell

One problem Friendster’s engineers had to address was optimizing the way queries were performed against the database. “Imagine asking a question to a database, and in that question, you ask 700,000 to 5 million questions to be answered all at the same time. That was what was being done to produce the social network part of site,” Pattishall says.

A key design goal for the new system was to move away from maintaining session state toward a stateless architecture that would clean up after each request, Lunt says. In addition, according to Pattishall, the team rebuilt the MySQL data in a more compact form, re-indexing and repairing tables in order to stabilize the site, improve page load times, and better handle traffic.

Another important step in stabilizing the site was getting more hardware in place and setting up a system to automate the hardware selection process. “Rather than buy big, centralized boxes, [our philosophy] was about buying a lot of thin, cheap boxes. If one fails, you roll over to another box,” Lunt says.

Friendster changed to a different white-box vendor, Open Source Storage.

“They gave us rock-solid, stable machines so we could skip time in qualifying the actual hardware itself,” Pattishall says.

According to Lunt, another key step to stabilizing the service was to install a monitoring system that could alert engineers when a system or hardware piece failed. For that part of the puzzle, Lunt chose the open source network monitoring program Nagios.

Creative configurations

With site stability within reach, Pattishall developed some creative MySQL tools of his own that wouldn’t have been possible with off-the-shelf technologies. To address the lack of management features available with MySQL, Pattishall developed a management tool that provides a real-time view of a cluster and that graphs database statistics across time. Pattishall is releasing the tool as an open source project.

Another problem was the management of the hardware, which proved labor-intensive; disks often failed, requiring manual replacement. Lunt and Pattishall looked for a better way to manage the systems, eventually creating a unique configuration consisting of 64-bit AMD Opteron hardware running on Linux with a custom kernel for talking to a SAN.

“The boxes connect over a fiber link, so if a box fails, we don’t lose all those disks and can get back up quickly. No one was using the configuration we’re using,” Pattishall says.

Pattishall also developed a custom configuration to load balance MySQL traffic using an appliance from NetScaler that was originally designed for accelerating Web pages.

“When our Web server makes a database [request], from the application point of view it looks like it is connecting to a single server. Because of the load balancer, we can take one request and put on 20 boxes. We are able to handle more load than if we stuck all those requests on a single box,” Pattishall says.

Development on the new version of the Friendster site running Apache and PHP was completed in June 2004. In the process, Friendster’s development team had grown to 20 engineers, primarily focused on rolling out new features to the service.

“Open source makes it easier to do that. The majority of new features have been built on open source,” Pattishall says.

According to Tim Denike, senior Unix administrator at Friendster, open source was the only solution to conquer the site’s massive growth problems. “Open source tools allowed us to scale a massively complex application across a system that required very little administrative overhead compared to those of other companies,” he says.

So the next time your company has to weigh its technology options for an IT project, don’t hesitate to throw open source into the mix. Mature open source technologies really are ready for prime time; the new and improved Friendster is living proof.