What's going on with Rogers?

[follower]

Network Disruption Triggers Internet and Phone Outages in Canada (Published 2022)

The outages affected customers of one of the country’s largest telecommunications companies. Some local police services said customers were unable to call 911.

www.nytimes.com

 

[Clark Griswald]

I know I'm late, but the issue has been fixed (as per google.news homepage).

7/14 Update: My apologies to all the people still effected by this "maintenance" (BS) issue.

 

[L&LD]

No, not fixed for everyone. They're still working on it. And they won't say what the issue was/is either.

 

[shabbs]

Holy crap that was a day. Got cell service back last night... home internet/tv/phone this morning. Seems fixed "mostly" but some areas still having issues and even some folks in the GTA losing internet again this morning.

Incredible video here showing the chain reaction and BGP route collapse...

Would love to know the real root cause, hopefully that leaks eventually.

 

[RMerlin]

And they won't say what the issue was/is either.

And that's where they truly failed this test.

Outages and issues are a fact of life with any technology. What is really important in my book is how it gets handled. Rogers failed miserably at communication - which is ironic considering communication is their core business...

Last year, my web hosting provider had a catastrophic failure within multiple datacenters. Took between a few days and a few weeks to recover (depending on the customer). Throughout the whole issue their CEO was communicative, at one point he was also posting daily videos where he would explain what had been done the day before, and what was the plan for the next day. I am still with them, and have no intention of moving in the near future.

Another exemplary case is Linode. They provide issue updates over email, and when it's an actual outage, it gets followed a few days later by a postmortem explaining what happened, why, and what steps they are taking to prevent it from happening again.

 

[shabbs]

Yeah the messaging during this whole event was terrible. First comms came over 4 hours AFTER the event started. Then the messaging all day was "We're working on it." Zero details or hints at progress, ETAs and direction. I get that ETAs are hard to put out - we do the same when Business is all over our butt when an app is having issue. Their CEO was even out there saying they did not know root cause. None of their messaging was reassuring. Inexcusable. Absolutely bungled it.

 

[RMerlin]

According to Cloudflare, the issue was related to BGP.

 

[RMerlin]

I get that ETAs are hard to put out - we do the same when Business is all over our butt when an app is having issue.

But saying "should be restored today" is still a useful piece of info they could have communicated. When that issue started, I had in mind an outage that happened a week ago with a Japanese provider, who was down for two days.

 

[shabbs]

According to Cloudflare, the issue was related to BGP.

Yeah - seems they lost all their routes - that video I shared above is incredible. Gut says they did some SW update and bungled that and then could not recover it or roll it back for whatever reason.

 

[shabbs]

According to Cloudflare, the issue was related to BGP.

For those curious, they provided a summary of the incident from their view. Very interesting.

https://blog.cloudflare.com/cloudflares-view-of-the-rogers-communications-outage-in-canada/

 

[Tech9]

Someone hacked them or they hacked themselves like previous time.

 

[dosborne]

Although our cell and internet services were back as of about 3am today, our Ignite TV service is still unwatchable, so not "back up and running". We are able to get a few frames of video every 10-15 minutes. This is both on the Roger's "box" and via their Ignite TV app (via cellular from a different part of the city) (Ottawa).

Wasn't willing to wait in the projected 6 hour queue time when I called in but did wait over an hour on the online chat function. I don't give a crap about a rebate (what, $5 each maybe), I just want *all* my services back. I'm so used to putting the news on over the last 2 years.....

At least I was able to get the big picture of what was happening by going to a local Tim's for free WiFi Friday around noon. Without internet (which includes VoIP phone), cellular voice and data and Tv service, we were essentially in a black hole LOL

i *almost* had to resort to knocking on neighbours doors to ask for a temporary WiFi feed as I can "see" a bunch of Bell WiFi in range, but that would have meant actually talking to actual people. Thank God internet was back before THAT scenario played out.

 

[Tech9]

Talking to actual people is horrifying indeed, but I wouldn't even think about going outside without knowing what the weather is.

 

[RMerlin]

Without internet (which includes VoIP phone), cellular voice and data and Tv service, we were essentially in a black hole LOL

 

[dosborne]

All your eggs in one basket

I did make it sound a bit more dramatic than it actually was. There are many free wifi locations we could have used, and my brother is an "all Bell" guy, so we could have gone there, or picked up a pay as you go card etc, in the end, we just watched some of the 15TB of tv we had previously stored on the NAS.

 

[Tech9]

I'm thinking about getting second ADSL line at home. TekSavvy again on Bell network.

 

[RMerlin]

I'm thinking about getting second ADSL line at home. TekSavvy again on Bell network.

My cell on Fido is my backup here. 5 hours of free data per month on top of my 10 GB plan, while my internet connection is TSI VCable. It was handy when my local cable cell went down for a whole day following a major fire near the local Videotron facilities (power was cut in the area for the day).

 

[shabbs]

I'm thinking about getting second ADSL line at home. TekSavvy again on Bell network.

Yeah buddy of mine has a Bell 30/4 secondary connection at $25/month for these situations with a fail over switch.

 

[shabbs]

Although our cell and internet services were back as of about 3am today, our Ignite TV service is still unwatchable, so not "back up and running". We are able to get a few frames of video every 10-15 minutes. This is both on the Roger's "box" and via their Ignite TV app (via cellular from a different part of the city) (Ottawa).

Same with us yesterday - Rogers Ignite was unwatchable. It looked to come back to "normal" late last night around 10pm and seems fine this morning.

EDIT: I lied - still having issues with a number of channels not coming in... one of them being TSN5 - need my F1!

 

[Tech9]

My cell on Fido is my backup here.

TekSavvy uses Rogers network for high-speed Internet plans in most of Ontario and Fido is a Rogers company as well. This won't work here.