https://www.smh.com.au/technology/telstra-s-only-two-timing-experts-had-been-sent-home-to-rest-before-network-fell-over-20260902-p60tmw.html
The only two Telstra engineers with deep expertise in the system that failed in July were on a mandatory break when the national mobile outage they had unwittingly triggered took hold, an external investigation has found.
The outage cut hundreds of people off from Triple Zero, halted trains in Victoria and NSW, and brought down eftpos systems across the country. It has triggered a Senate inquiry and an investigation by the communications regulator that could result in Telstra being fined up to $30 million.
The pair had replaced a faulty power supply in a timing chassis in Melbourne in the early hours of July 8. When it came back online at 2.50am, a GPS card inside reset and began telling the mobile network the date was 2006. The card was missing a firmware update for a predictable fault the supplier had already flagged in bulletins, confirming this masthead’s (the Age), July reporting.
The report by Technology Audit Partners found alarms on the timing servers were checked only during business hours by a small number of staff and did not appear in the monitoring tools used by round-the-clock support teams. When errors began to occur, nobody on duty could find records of the servers, the work done that night or the people to call.
Chief executive Vicki Brady said the company accepted every finding.
....
Brady also revealed Telstra had missed eight failed Triple Zero calls during the outage and never carried out the welfare checks required to find out whether the people who made them were safe. She blamed a manual processing fault and a monitoring fault.
....
“It shouldn’t have happened. We should have picked those eight up,” Brady said. “We know how important Triple Zero is.”
....
The telecommunications company also revised the start of the outage to 2.50am on July 8, almost 50 minutes earlier than it had said. The previous time reflected when a maintenance ticket was closed rather than when the fault began.
Brady said the failure was “not due to infrastructure failing” but to an undocumented design change made in October last year and a software update that was never applied to a GPS card.
....
“It’s not about not enough resources. It’s not about not enough money,” she said. “It’s about the prioritisation.”
....
The Australian Communications Consumer Action Network said the report showed a governance failure rather than a technical fault beyond Telstra’s control, and renewed its call for enforceable reliability standards.
“We rely on telcos to ensure that their networks are managed with the seriousness commensurate with public safety,” chief executive Carol Bennett said. “This report shows that didn’t happen. Not because the problem was unforeseeable – in fact it was very much foreseen – but Telstra didn’t treat it as a priority until it caused a national outage.”
....
More than 30,000 customers have contacted Telstra for compensation, and it has paid just under $1 million in credits, an average of about $15 for consumers. Brady urged anyone still out of pocket to come forward.
“Modern networks are complex but complexity is not an excuse,” she said.