Hush! Call that bug 'severe' at your own risk

After an acquisition, the new management team does all it can to play down and whitewash security events

Hush! Call that bug 'severe' at your own risk
Credit: Guilherme Tavares

Ever had a job where you had to constantly look over your shoulder and were hesitant to make decisions because you could be blamed for them? It's not fun. I once found myself in such a situation, but fortunately, it didn't last too long.

In the late aughts, I worked for a large national bank that was acquired by an even larger national bank. Our bank still ran our own IT operation, but we engaged with the larger bank's incident management team for any major problems. Part of my job supporting our Retail Bank organization was incident coordination.

The larger bank had a severity ranking for incidents that was somewhat similar to what our bank had used, but took it a step further and defined the differences in severity levels using hard numbers of dollar impact and the amount of customers affected.

Severity 1 was the highest level, which meant the financial impact was greater than $100,000 or affected at least 10,000 customers. Severity 1 could also be invoked for any issue that "might make national headlines," such as a security breach exposing confidential customer data.

Who you gonna call? Upper management

The system seemed straightforward on paper, but I started to notice an undercurrent of fear among employees toward senior managers. An early example jumped out when I needed help assessing the impact of a sporadic issue. At the time, we were being introduced to the incident severity rating scheme; we knew that some customers were affected, but we couldn't provide a good estimate of how many, so weren't sure what severity to list the problem under.

When I asked for help, no one seemed to want to make the decision on the severity level. I heard things like, "Well, we could do this, but then upper management might disagree." They were afraid of being second-guessed.

I was eventually instructed to contact a senior manager for the final call, which resulted in a delay of about two hours because that manager was in transit. He called back after his plane landed and seemed puzzled that I was asking -- but I explained that I had talked to several other people in the organization who did not want to make the decision. This manager cast the vote, but it took time and involved a senior manager, all for a decision that technically could have been settled on a lower level.

Obfuscate your way to success

One day we had an issue with one of our ATM switch partners (an external vendor) that affected our customers' ability to get credit and debit card transactions approved. I quickly realized we had a Severity 1 incident. It met the criteria: We estimated that around 30,000 customers had been affected, and it had the potential to impact far greater than $100,000. We had as close to hard numbers as we could get, and it seemed very cut and dry.

I joined the conference call to report that this issue was Severity 1, but was told, "No -- this is only Severity 2." I asked why, when it filled all the criteria for a Severity 1. An incident manager for the larger bank asked me to call him after the meeting on his cellphone so he could explain.

When I followed up, he told me he agreed that "on the surface" this could be considered a Severity 1 issue, but he said I did not take into account that they estimated only 10 percent of affected customers would actually call the bank to complain. The estimate of "30,000 customers affected" was changed to 3,000.

I told my bank's management how this decision was made so that they would not be blindsided if anyone questioned why the incident had been lowered to Severity 2. I was relieved that there was no fallout from what I thought was outright dishonesty on how the incident's impact was rated.

A fearful, cowering culture

As time went on, I discovered that this kind of practice was the norm. The larger bank had such a culture of fear that its IT teams did all they could to avoid labeling an incident as Severity 1. Whenever a Severity 1 issue came up, senior managers immediately wanted to know whom to blame. The team would rather push the boundaries in the paperwork than be put under such intense scrutiny.

Not all Severity 1 incidents were similarly downgraded. For example, major outages like online banking going down could not be swept under the rug. The ensuing postmortems were certainly uncomfortable.

As a result, the tech teams mostly took care to avoid dings from senior management, who in turn wanted to protect the business's image first and foremost, then lay blame if the public profile took a hit. I had to grin and bear it for about a year until we converted our accounts to their retail bank system.

It was at this point that the company laid off most of us in IT. However, I was glad to get away from a business that fostered such a culture of fear and showed distrust in its employees.

From CIO: 8 Free Online Courses to Grow Your Tech Skills
View Comments
Join the discussion
Be the first to comment on this article. Our Commenting Policies