The Olympics network: Faster, stronger -- and redundant

Yan Noblot, information security manager at Atos Origin, says the key to that is to build in redundancy -- and lots of it. "We have doubled everything, because we need 100 percent availability at games time," he says.

Free IT resource TechNet: More ways to know it, share it, and keep it running. Sponsored by Microsoft Free IT resource Attend the SOA Executive Forum: Breaking SOA Bottlenecks SOAExecForum.com/may2007 Sponsored by InfoWorld

And when he says everything, that goes for the routers and switches at each site, the datacenters that process the results, even the PCs on the desks in the control room.

An SDH (Synchronous Digital Hierarchy) network composed of two STM-1 rings supplied by Greek carrier Hellenic Telecommunications Organization (also known as OTE, an acronym based on its name in Greek) links the Olympic venues at 155Mbps. "We only use one ring, the other is for redundancy," Noblot says.

The 36 competition venues and 20 or so noncompetition venues are linked by fiber to the SDH ring. All the venues are connected by two different routes, with a ring for each venue, and traffic from each venue is served by two different telecommunication centers, according to Elpida Trizi, a spokeswoman for OTE. "We've constructed the network in such a way that we're able to provide a service even if one of the routes is damaged," she says.

Event results and data from the games management system are stored in two datacenters hosted by OTE, which also supplies the SDH network. The primary datacenter is located near OTE's headquarters in Marousi, just across the main highway from the Olympic stadium; the other is another several hundred miles away, still in Greece but in a different earthquake zone.

The datacenters sit directly on the SDH ring. "One reason we did that is because we do real-time replication between primary and secondary datacenters," Noblot says.

That direct connection is made through a pair of Cisco 7200 series routers. "We have two of each, at least, for redundancy, configured in such a way that the traffic would be automatically rerouted," Noblot says. "Behind this we have two Catalyst 6513 layer-3 switches running services. We use it to do VLAN routing. We also have a firewall and intrusion detection system in the same chassis."

To keep things orderly, Atos designed three different LAN configurations: one for the largest venues, including the Olympic stadium and the water sports center; another for midsize venues such as the equestrian center; and one for the many smaller venues.

Each venue has a pair of access routers, with models depending on the venue size, and behind that a pair of switches to send all the traffic to the intrusion detection system. After that comes the distribution layer, a pair of switches for VLANs and routing between the VLANs. "The very last layer is the access layer where we have Cisco switches that plug directly to the servers and the PCs," Noblot says.

Segmenting Traffic

Atos is using VLANs both to simplify troubleshooting and to limit damage if anyone manages to break into the network. There are separate VLANs for the commentator information system, information diffusion applications, and the game management system. Technical services, directories, management and monitoring, and the on-venue results system each have their own VLANs too, sometimes several per venue for the same function.

"The purpose is to segment the traffic so we can monitor it and contain potential issues," Noblot says. "If someone brings in a virus, that would be contained on systems on the same VLAN."

Software distribution is another service secured over a VLAN. Atos is using Symantec's Ghost and software from LANDesk to remotely load software onto PCs.

That's an issue Noblot has kept in mind since an incident occurred at a practice event in Greece. "We were just setting up for the mountain bike event, and we had a hardware failure," he says. "We said, 'Just send a guy from this room with spare hardware,' and the guy called up and said, 'Are you nuts? It's not possible, it's two kilometers uphill!' "

In this case, the team ended up rebuilding the defective component's software over the network.

Anything that can avoid the need for a time-consuming journey to make hands-on repairs is welcome, because the SLA specifies a 10-minute resolution time for severe problems. "That's not to take ownership of the problem, but to fix it," Philipps emphasizes.

What makes the Olympic Games a unique project is that the athletes aren't going to stop running just because the server has. As Philipps says, "When we speak about fixing something, it might be a work-around, a decrease of functionality, but the key thing is the show must go on."