“Technology outage” brings NS operations to widespread standstill; impact to last weeks

Yikes. Norfolk Southern’s year to forget has another unfortunate reason to be remembered. A NS “technology outage” today pretty much brought operations to a standstill — and the company states the impact on those operations will last “a couple of weeks”.

Story from Trains: https://www.trains.com/trn/news-review ... er-trains/

Lead sentence:

A technology glitch that affected several Norfolk Southern systems — including positive train control and its freight yard operating system –— caused delays across the railroad today.

NS news release:

This morning, Norfolk Southern experienced a hardware-related technology outage that impacted rail operations. At this time, we have no indication that this was a cybersecurity incident. Our teams worked throughout the day and successfully restored all systems at 7:00 p.m. ET. We are safely bringing our rail network back online. Throughout this, we have been in contact with our customers and will work with them on updated timing for their shipments. We expect the impact to our operations to last at least a couple of weeks.

You can expect more and more of this in the future thanks to the PTC mandates. Where three years ago you could still run an operation based on 100 year old wayside signalling technology, now there's zero room for an outage when the systems feeding the magic box fail.

Every time you add new links to the automation chain, you add new points of failure...



Sent from my SM-S911U using Tapatalk

Has happened several times lately on the NECR, Amtrak taking the biggest hit.

Don't know that I believe anything NS is selling.

And another one. Supposedly things were down from Friday (9-29) evening to 1:30 Saturday morning. But again they are saying it will be at least 2 weeks before the effects are mitigated. Really? Things halt for about 8 hours, and it's going to take 2 weeks to clear the backlog? What kind of railroading is that?

One might wonder if things on NS are so bleeped up that they engineer "outages" to give them an excuse to blame.

With regard to the more recent incident in late September as noted by BR&P, this is a bit late on my part, but NS says it was not related to the August one. Herewith a portion of the NS news release:

"At this time, the company has found no indication that this was related to an unauthorized cybersecurity incident, or the outage of August 28. We are continuing to examine the root cause of the incident but believe at this time that it relates to a vendor product defect. Further, Norfolk Southern has undertaken a total review of the resiliency of its data center and network technology to prevent future outages."

BR&P wrote:And another one. Supposedly things were down from Friday (9-29) evening to 1:30 Saturday morning. But again they are saying it will be at least 2 weeks before the effects are mitigated. Really? Things halt for about 8 hours, and it's going to take 2 weeks to clear the backlog? What kind of railroading is that?

One might wonder if things on NS are so bleeped up that they engineer "outages" to give them an excuse to blame.

Depends on what they are referring to when they say mitigate the outage. I can see a scenario where they were loading future date operational data and something got messed up, and it's easier to Simply let the missing data restore over the course of two weeks then it is to destroy data that had already been updated during the outage.

Without having actual context around what was down for 8 hours, anything is possible.

Sent from my SM-S911U using Tapatalk