Facebook, Instagram, and WhatsApp Were Down in Major Outage
It’s not just you: Facebook, Instagram, WhatsApp, Messenger, and Oculus VR are currently down. The Facebook-owned services all use a shared infrastructure and are unreachable in multiple regions since about 11:41 AM eastern time.
There do not appear to be any outages that can be linked to internet providers or service centers like Amazon Web Services (AWS), leading some to speculate that the issue is specific to Facebook and its network.
Down Detector, a user-based reporter, shows a large spike in outage reports in the last half hour which indicates that the problem is widespread. Down Detector also shows simulataneous spikes in downtime for the Facebook entities.
Andy Stone, a member of Facebook’s communications team, says that the company is aware that some people are having trouble accessing its apps and products and that it is working on getting things “back to normal” as quickly as possible.
We’re aware that some people are having trouble accessing our apps and products. We’re working to get things back to normal as quickly as possible, and we apologize for any inconvenience.
— Andy Stone (@andymstone) October 4, 2021
As noted by The Independent, Facebook has often been cryptic when it comes to explaining outages. In 2019 when it was asked to respond to its largest outage in years, the company says that it had “triggered an issue” during “routine maintenance operations.”
In a leaked transcript published by The Verge in 2019, Mark Zuckerberg said that outages are a “big deal” and that it can lead to its user base clamoring to competitors instead. It can supposedly take “months” to fully win back the trust of users back to the platform, if they come back at all.
Facebook and Instagram have faced mounting pressure from Congress and advocacy groups for a set of reports that show the company’s own internal research confirms that its platforms are harmful to teens. The company says that those claims were taken out of context, and published those referenced internal reports publicly last week.
As you might expect, outages at Facebook are extremely costly: the company reported revenue of about $86 billion in 2020. That’s about $2,700 in revenue generated every second. It remains to be seen how many seconds in lost revenue this current outage ends up amounting to, but the final figure could be quite large.
Update: Nearly five hours after the outage was first reported, Facebook and its entities remain offline. According to sources at the company who spoke to the New York Times, the actual root cause of the issue is still unknown. Two members of the security team say that it is unlikely that it was caused by a cyberattack as if it were, it would be unlikely to have affected so many apps at once.
Some experts believe the problem most likely stems from a misconfiguration of Facebook’s servers which can usually be fixed by rolling back to a previous configuration. The fact that this issue has not been corrected yet, though, seems to indicate that the problem is more complicated.
Someone on the Facebook recovery effort has explained that a routine BGP update went wrong, which in turn locked out those with remote access who could reverse the mistake. Those who do have physical access do not have authorization on the servers. Catch-22.
— Steve Gibson (@SGgrc) October 4, 2021
Some reports also state that the issue did not just affect Facebook’s web services, but also its security system, as noted below:
Was just on phone with someone who works for FB who described employees unable to enter buildings this morning to begin to evaluate extent of outage because their badges weren’t working to access doors.
— Sheera Frenkel (@sheeraf) October 4, 2021
Workplace, Facebook’s internal communications platform, is also offline, leaving most of the company’s employees unable to do their jobs.
Update #2: Facebook started coming back online Monday afternoon, though the company cautions that it will take some time for all of its apps to stabilize. One other effect of the outage that many may overlook was that millions were unable to use Facebook to sign into many other apps and services ranging from shopping sites to thermostats, as noted by the New York Times.
*Sincere* apologies to everyone impacted by outages of Facebook powered services right now. We are experiencing networking issues and teams are working as fast as possible to debug and restore as fast as possible
— Mike Schroepfer (@schrep) October 4, 2021
Them fighting words… but it does feel like a snow day.
— Adam Mosseri 😷 (@mosseri) October 4, 2021
Cloudflare’s chief technology officer John Graham-Cumming speculates that based on how the social media site was reacting during the outage, it was likely due to a misconfiguration of Facebook’s servers that caused the problem. The New York Times says what happened was akin to removing people’s phone numbers from under their names in a smartphone’s address book, which would make it impossible to call them.
“It was as if Facebook just said, ‘Goodbye, we’re leaving now,’” Mr. Graham-Cumming told the Times.
Getting Facebook’s services back online required the dispatch of a small team to Facebook’s Santa Clara, California data center to do a manual reset of the company’s servers, a move that appears to have worked. How the misconfiguration of the server happened in the first place is still has not been confirmed by Facebook.
