I happened across this breathtaking story in Ars Technica last week that details a discovery Brad Reese made many months ago. This didn't get much attention at first, but it appears to be blowing up now. The story dovetails nicely (however sadly) into my discussion last Monday of the need for early IT involvement in IT projects.
To boil it down, West Virginia overspent about $5 million in federal money when designing and building a statewide network. Instead of specifying and purchasing adequate hardware for each location it needed to serve, the state bought a $20,000 Cisco 3945 router for every single location, regardless of size or capacity. That's how it wound up with a library the size of a doublewide that's open only three days a week, yet boasts a Cisco 3945 for its meager Internet connection. I imagine that if West Virginia had gone with more suitable hardware, it would have had enough money left over to keep the place open all week long.
[ Cash in on your IT stories! Send your IT tales to firstname.lastname@example.org. If we publish it, we'll keep you anonymous and send you a $50 American Express gift cheque. | Get the latest practical data center advice and info in Matt Prigge's Information Overload blog and InfoWorld's Data Center newsletter. ]
This is what happens when nontechnical decision makers ask vendors to design their networks and send them the quote. No matter how much Cisco denies it padded the specs or claims the routers were necessary for "future expansion", the company can't change the facts. Any network architect worth their socks knows this is a huge load of BS; moreover, so does Cisco.
No WAN this size -- absolutely none -- has identical requirements in every single location. Further, the Cisco 3945 is billed in Cisco's own literature as a solution for medium-to-large installations. By any possible measure, it's egregious overkill for nearly every location in the West Virginia network in which it was deployed. Many of those sites would've been perfectly served by a small branch router costing less than $1,000.
The cost of this baffling choice was gauged at $5 million in the article, based on the assumption that the locations could have been served by $5,000 Cisco 2800- or Cisco 2900-series routers. If we push the reality check even further and consider that $1,000 to $2,000 routers would have done the job, the waste is potentially much larger.
There's no doubt this is a boondoggle and mistakes were made at every level of the deal. No amount of "futureproofing" or "forward thinking" can dislodge the fact that there is no rational reason for this purchase to have been made and, more important, this quote to have existed at all. This is a direct result of predatory salespeople and (at best) apathetic state officials. At worst, it's yet another example of backroom dealings so over the top that they've actually found sunlight.