I know from experience that 75% of what I’m told is fake news that’s been washed through 8 layers of management. My question is why did Wells Fargo blame the outage on a fire when the fire department and other sources said it was caused by dust from construction. Is the dust story even real? Why would Wells Fargo not just admit it was dust. Somethings fishy and I want to piece it together.
10 replies (most recent on top)
The next test is April 25th... you all know what I'm talking about, Saul mentioned it..... be ready......
@3esi, they don't get commended because we are an IT department, we are not islands.
A datacenter failed, that's on all of us.
If those folks were raising red flags for the last, oh, I don't know, decade or so, then we can say they were doing their jobs.
It's a bank wide IT failure of epic proportions.
Real IT people know this. I think the bank has been too insulated for too long, and saying "my stuff worked" is not an excuse.
S4!+ happens, no doubt, but this was on all of us.
Why shouldn’t the teams whose apps stayed up during a primary data center outage not be commended? That means they did architect and implement correctly, and (gasp) even test their BCP plans, even if exec managers didn’t mandate it.
Someone said some apps didn't fail and those managers should be commended?
You don't get a medal for blind dumb luck, if you're not testing your BCP and backups you are failing at IT.
The end.
SMH.
We spend $$$$ on redundancy, but that DC outage was the first true test. Many apps failed over properly and those app (mid level) managers/owners should be commended. Other systems failed miserably, but it is being treated as a learning experience rather than a basis for termination.
That we have not been testing our failover processes and BCP plans is one of the biggest IT scandals ever.
Period.
They don't know what they're doing and the CTO got fired for it end of story.
The answer to the below question is very simple- extraordinarily bad management. This is why the company is in a boatload of regulatory trouble.
Raising concerns or asking questions or suggesting improvements was something managers actively discouraged. The proof is in the occ report. It’s starting to change, but really slowly and most managers still prefer the ostrich approach.
At the first big IT company that I worked at they had regular tests at the disaster recovery data centers.
Why did this not happen at the much, much bigger company Wells Fargo? They should have been
able to quickly switch to the backup disaster recovery center with a very short outage time.
Data center went into power down, power down caused by activation of fire suppression system, fire suppression activated by maintenance activities that created airborne particles. Not a conspiracy, but failovers to back up did not work so a minor snafu becomes a disaster.