Some lessons are more painful to learn than others. My pain point hit its peak when I had the misfortune of being on a team implementing a new ERP system that was, to put it mildly, fraught with errors from beginning to end.
I had been at "Company W" for nine months when the bosses formed a committee to search for a new ERP system. After six months, they settled on one that was highly customizable, which Company W wanted so that it wouldn't have to change much of its home-grown ERP application.
As it turned out, the project had a major stumbling block: The customization specs were not fully expressed to the ERP vendor.
The project players
"Tom" had become the ERP project manager for Company W several years before. He was really a database administrator, but he got the ERP job because no one else wanted it at the time. In addition, Tom had no project management experience.
The ERP vendor also had a project manager, "Ben." Whenever Tom, the ERP committee, and Ben would meet, Ben would ask Tom and the group, "Have you validated your assumptions?" This statement became a joke at Company W, but it was a legitimate question and no one ever answered it.
Tom looked at all of the hardware and software requirements and decided to give the different tasks indiscriminately to each of the IT staff. For example, one person was responsible for the network/firewall/switches/routers, but one of my tasks was figuring out what network and network devices upgrades we needed.
Following the information I was given from Tom, I found that more than 90 percent of our network devices needed to be replaced. I passed my list on to Tom. He purchased about 10 percent of the items for the project, despite requests from me and other IT staff members.
I set up the storage and server hardware with the operating system and was ready for the application software and the database software for a pre-production and production environment. This involved multiple servers, and I followed the recommendations of the vendor. Once the applications and databases were installed, it was time to do the initial testing.
The results were very discouraging: The ERP client application was at least 10 times slower than the old one, even without users were working on the system.
All signs point to chaos
Tom decided that every department should have its own test database for its own purposes, separate from the other departments. But he never made plans for integrating the departments in the pre-production environment for end-to-end testing. Another disturbing issue: Tom found out many items did not work properly, but he was waiting to fix them after the project was completed. The list was getting quite lengthy.
But the project marched on. Tom never posted any timelines or gave any of the departments due dates for their tasks. He relied on Ben to keep everyone updated, but this gave Ben a false indication of the ERP project's progress. Ben got approval for using the vendor's consultants, and he had them customize the program as best they could with the few specifications Tom had provided.
After several months, Ben thought the project was almost complete, but Tom would say that this or that department hadn't finished testing, though it was supposed to have been done by the timeline that Ben sent to everyone on the ERP committee -- the only one we saw until the very end.
After a year, the consulting company did not get the program customized the way Company W wanted due to these breakdowns in testing and communication. But Ben still wanted to close the project as completed because there was nothing left to be done on his end.
After a year and a half, Tom convinced Company W's upper management that they no longer needed the vendor's project management consultants. Instead, Tom found his own, and they took over the customizations.
Almost two years after purchasing the software, upper management was getting impatient and informed Tom that the application would have to be ready at the beginning of the new year. All this time, none of the departments were given clear guidance on what they were supposed to do, and there was never any end-to-end testing (from order, to payment, to shipping, to billing) of the app. On the client workstations, the new system was still 10 times slower than the old one.
Day of reckoning
Tom set the date for rolling out the new system for the week after New Year's. Three days before the deadline, Tom assembled all of IT for a meeting on implementation. For the first time, Tom showed us a project management timeline of his own. However, it had been backfilled with information about timelines from the past and other items that were supposed to have been accomplished up to the current date. Nobody had seen this before.
More disturbing, the spreadsheet that went along with it had more than 119 (and counting) fixes that needed to be done after the system was implemented. When I asked Tom about the slowness of the ERP client, Tom replied that he thought everything would be fine after implementation.
The implementation deadline came, and the problems were many.
For processing, orders were taken and went as far as printing the packing slip, but there it stopped. Staff couldn't scan in items properly and had to enter them manually. Several departments were not able to use the system, and it was very slow for everyone.
For the next six months, they had to hire double the staff for the warehouse and had shipping working four to eight hours of overtime during the week and on weekends. In accounting, staff would spend four to six hours during the weeknights invoicing, after putting in an eight-hour workday.
With minor changes, we were able to reduce our server reboots during the business day down from three days a week to less than one. During that time, Tom would usually blame the problems on the storage system, the servers, or the network. I worked with our storage vendor to make sure our SAN was working optimally, and I fixed minor configuration issues that were not causing problems.
After going through all of the servers, I could not find anything wrong with them. I started to look at the application itself. I was able to see the symptoms, and going through logs with the help of the vendor's support, I found the right log files to examine. The problem was poor customizations from both teams of consultants.
I showed Tom the logs, but he didn't want to believe it and did nothing to resolve the issue. This became an ongoing battle between us, and when Tom would approach me with the problems we were having with the ERP system, I would tell him the customizations needed to be fixed.
Over time, I managed to add the rest of the 90 percent of new hardware, updates, and other changes that Tom wouldn't originally let me do.
Also at this time, the vendor's user group blogs reported similar issues and pointed out poorly written customizations as the first place to look for application slowness. Some of the blogs also mentioned that the fewer the customizations, the faster the ERP application would run. Tom did nothing other than blame the servers and the network again.
A year and a half after we implemented the production system, Tom decided to look at the customizations because he was once again getting heat from upper management. Amazingly, Tom found and started correcting some of the sloppy and poorly executed cases. All of a sudden, the system was working almost as fast as the home-grown software.
After the performance improved, Tom was given superhero's status throughout the company along with a paid week off. When Tom talked to me about it (expecting me to compliment him), I reminded him that we had known about the customizations issue for more than a year before he started looking at it. Tom stopped smiling and changed the topic.
Eventually, the kinks were worked out, but it took way too long. At least those of us paying attention learned a lot of lessons in what not to do.