Had a customer meeting today and he said he won't buy another Cisco product again until the software quality is proven again. Apparently tons of bugs are causing issues all over the place. Anyone else hearing this?
17 replies (most recent on top)
If quality wasn't an issue, it wouldn't be a top priority of every BU and DE exec.
It should make you wonder why this has been the case for decades and yet the quality hasn't improved. Directors and Vice Presidents are often asked if all the processes which are slowing things down have resulted in quality improvements and they reply "no, it's actually getting worse, but we have to keep doing the same thing we're doing until every problem is fixed and then we can look at how to change things for the better" as if infinite breakage will fold the universe inside out and everything will suddenly work.
You missed the other top priority of "becoming a systems company instead of a box company. Do you want to buy a box?" That gets me every time! Systems Engineering is a discipline for which there are plenty of graduate programs. I couldn't name five people at Cisco who could pass the first quiz in the first class in such a program. Hint: there are no funnels, tootsie rolls or plungers involved. Requirements? Design? Modeling? Interface Definitions? These things are foreign to almost everyone I worked with at Cisco, and the complete lack of these skills at the leadership level is why you are spending most of your time trying to fix things rather than actually adding real value.
And I've watched my engineering move from US to India over the last 10 years. I could write a book on the differences in engineering culture between the two shops.
This is crap. Run any decent code quality metrics on any of the code bases at any point in time and you'll find it's all bad. Really bad. Many called out major hardware faults early on as well but were ignored until those faults became showstoppers and stalled programs for years.
I could write tomes on all the failings in both hardware and software dating back to far before India was a major part of the development process. I'm not talking about simple mistakes, I'm talking about things so insane I would never believe them if I hadn't been there to see them. Cisco created many worst practices long ago avoiding things which have been well understood for millennia (see "functional decomposition") and the Indian teams adopted these worst practices with a vengeance. Pot, meet kettle.
And the documentation...
Yes, the documentation. Like API and Reference Guides which were never written for the assorted operating systems. Like the many SFS and SDS documents which are nothing more than templates with the title, author and EDCS document number filled in. Like the test plans with no steps to reproduce nor any pass/fail criteria. Like the plagiarized white papers which don't actually say anything. Like the external documentation which was well written for each platform but because you had so many release branches and no internal definitions each time code from one platform which changed shared code was collapsed and synced to another it would break the second platform's externally declared behavior, which would be fixed and pushed up and down back to the first platform which would break its externally declared behavior, which would... It's like in 2000 when whole teams were adding platform_may_suspend to deal with CPUHOGs and other teams were removing those same calls to fix the race conditions they introduced which would result in CPUHOGs, which would... If it weren't for the need for customers to keep pumping in cash to keep these things going Cisco would have many real patents for perpetual motion machines.
Welcome to Agile. Just a way to push out sh!t at breakneck speed. I'm sure Agile Development has a place somewhere, when done right. What it appears to be @ Cisco is just a way to never mark something 'done' and allow the business an open license to never get their act together.
If quality wasn't an issue, it wouldn't be a top priority of every BU and DE exec. Check every DE exec presentation or all-hands agenda and quality is at the top of all of them. It's the reason why you cannot get a feature on a roadmap (their view is that features cause bugs - less features = less bugs). Quality is our number one issue along with not being able to trust roadmaps any more (which DE blame on having to spend resources fixing quality).
And I've watched my engineering move from US to India over the last 10 years. I could write a book on the differences in engineering culture between the two shops. It's not just complexity, the whole mindset is different - and not in a good way. Features that are claimed to be "supported" are never tested (dealing with one of those now - it will never work - so how COULD It have been tested?)
And the documentation - again, it's a change in mentality, and now the documentation is just woeful, when 10 years ago, we had a great reputation for good docco.
With every one IT manager saying not to buy Cisco, there are 100 s---ers who would. Cisco is here to stay no matter you love or hate its quality. It is no matter if the piece of junk is made by Americans or Indians. However, I'd like to say the Indian SVP / EVP in Engineering are lucky son of gun. They get to harvest the hard work done by Americans who were driven out of Cisco.
I was on the train yesterday and some IT manager was screaming down the phone at Cisco TAC due to an outage on his WAN because of an unidentified SW bug. I mean he was going Micheal Duglous in Breaking Down on the TAC guy. Some standout comments were "you've cost us hundreds of thousands today and you're telling me you don't know what caused this. Did you not build this f#cking cr@p?" and then "I'm going to make it my mission to ensure we never buy Cisco again". This was on a busy train in NY and everyone could hear. I almost approached him, but thought better as he had the rage.
Main killer of quality is not who does it but software sprawl and more complex hw. In mid 90s a interface card with 2x1ge was state of art....today its 2Tbps....hw has become exceedingly complex and so have the drivers. Sw is a mcmansonian sprawl of 100 comps all under different managers. Things break inadvertently all the time as people can no longer fully grasp the side effects of benign looking changes.
To all those crying wolf about india india, i will have you know i was the only brown in a group of some 30 pure snow white americans all veterans of the best univs and past cos in late 90s working on a router with far simpler hw and small tighter sw and our quality still s---ed big time. Half the time all seniors were in war rooms and cap calls.
Nobody has the time or aptitude to simplify things now. Layers get added to already muddled stuff and the pet soln is to add more debug cmds...already 1000s exist..we need to prevent problems.
Lately i see a turnaround in upper mgmt thinking. Automation and closing gaps in Ut and Dt is being emphasised strongly and directors willing to take a hit on feature velocity for that. Without stability the features are useless as none will buy.
I would love to see any evidence that quality was great 15 years ago when everything was done out of san jose by the self proclaimed blue bloods...
"Had a customer meeting today and he said he won't buy another Cisco product again until the software quality is proven again. Apparently tons of bugs are causing issues all over the place. Anyone else hearing this?"
You're either very new, or new to the field. We have issues with our Nexus 7k's not properly saving running configs. Bad power supplies for 4500 switches. Ports on Nexus 5k's just deciding not to work anymore. Other issues too numerous to mention. The only thing that you have going over our other vendor (Avaya) is that they just declared bankruptcy.
Last visit to India so called engineers literally sleeping the day away snoring and all.
For decades. You clearly aren't paying attention as it's raised at many all hands, at every customer address of the development teams, CAP cases, discussed here extensively, etc... You've clearly never been a customer during that time frame either.
Despite all this many competitor's products are far worse so we still buy Cisco for some applications.
its true, we outsource engineers and software to india, they have no idea how to do superior work, just do the min for the min pay. only a few senior people are keeping it together, its so bad we just leave them alone and do all the work between a few instead of having them further ruin the brand.
"Whats this got to do with layoffs?"
Poor quality greatly damages the Cisco brand and drives customers to other vendors. One bad product can change the customers "All Cisco in the network" approach and open the door for competition from all vendors for every network component. This is one of Cisco's biggest risks. Reduced sales and market share generate additional cost reductions and layoffs.
Customers pay a stiff premium for the Cisco brand. They expect to get superior quality to justify the extra cost.
Whats this got to do with layoffs?
Troll
which is the product , can you be specific ?
Correlation to the movement of product engineering to India and the hiring of fresh outs needed to survive.
There are no bugs at Cisco only undocumented features!
Cisco stuff has no bugs... the employees are the bugs... only way to get rid of bugs is get rid of the employees...