Here are two facts about Brook Colangelo's job as the CIO of the Executive Office of President: On taking his new position on inauguration day, he and his staff put in 80-hour weeks, "if not more." And in his first 40 days on the job, the White House email system was down 23 percent of the time.
Colangelo began his job on Jan. 20, 2009, the same day that President Barack Obama started his. On that first day, Colangelo walked to the White House, found his office with some difficulty, and then "delivered the first presidential Blackberry," as well as handhelds to all the top administration officials. "It was just a mind blowing experience," he said.
[ Keep up on the day's tech news headlines with InfoWorld's Today's Headlines: Wrap Up newsletter. ]
But Colangelo quickly realized that the White House's IT assets "were in pretty bad shape." Over 82 percent of the White House's technology had reached end of life. Desktops, for instance, still had floppy disk drives, including the one Colangelo delivered to Rahm Emanuel, Obama's then chief of staff and now Mayor of Chicago.
The White House CIO office had one data center, said Colangelo. "We had no redundancy," he said, before attendees of Computerworld Premier 100 IT Leaders Conference in Phoenix.
The problems became apparent on Jan 26, six days after the administration was sworn in. "Our email servers went down for 21 hours," said Colangelo. "In my professional career, there has not been a worst day since or ever."
At 5:30 a.m. on the morning of that already long outage, Colangelo was called over the West Wing to brief chief of staff Emanuel about it. "I was walking [to the meeting] with some other leadership. It was pitch black and I haven't gone home, and then the most amazing thing happened," said Colangelo. "As my two feet hit the door of the West Wing, my Blackberry started to buzz. I normally hate that feeling, but I got to tell you it was the best feeling I ever felt."
The email was back up. By the time Colangelo reached Emanuel's office, the meeting with the chief of staff was no longer needed.
Colangelo said the White House technology situation called for a massive review of technology, people, and processes to determine the situation.
The White House faced three or four more outages in the next 30 days or so. "When it was all said and done, in the first 40 days of the administration we were down 23 percent of the time," said Colangelo.
The White House CIO provides all unclassified services to nearly a dozen business units, including, the Office of Management and Budget and the Office of the U.S. Trade Representative. Some of Colangelo's previous jobs included working as the CIO of the Democratic Convention Committee, and IT project manager for the American Red Cross Hurricane Recovery Program.
As part of the process to upgrade White House IT operations, Colangelo held town halls with their customers, some whom were angry. "They had floppy drives -- I knew what they were going to say," he said.
The White House needed to replace much of the technology, including the email systems and storage area networks. As part of the effort, the White House created new positions, including the "GOALIE," which is the Government Operations and Lead for Inspection and Execution. GOALIE employees worked 12 hour shifts to give 24 hour coverage of the data center and manage any crisis. Previously, the data center was only staffed from 9-to-5.