Facebook explains the reason for yesterday's 6 hr outage

Facebook engineers have posted a preliminary analysis for the reasons behind yesterday’sunprecedented 6-hour outage which disrupted the online lives of billions of people, who were not only unable to share life events, but in many cases unable to communicate with loved ones via services such as WhatsApp and Facebook Messenger.

Facebook said that configuration changes on the backbone routers that coordinate network traffic between their data centres caused issues that interrupted this communication. This disruption to network traffic had a cascading effect on the way their data centres communicate, bringing our services to a halt.

Facebook says the underlying cause of the outage also impacted many of the internal tools and systems they use in their day-to-day operations, complicating our attempts to quickly diagnose and resolve the problem.

According to other reports many Facebook engineers were locked out of server rooms and buildings as their entrance credentials could not be verified.

Facebook employees can’t enter the headquarters because their badges don’t work, and those already inside can’t enter various rooms because access is linked through the IoT (Internet of Things) and so goes through the same DNS routes that no longer exist:#FacebookDown pic.twitter.com/8hAea9ZG4l

— Leah McElrath (@leahmcelrath) October 4, 2021

One report even states Facebook employees has to resort to angle grinders to get access to their servers, though Facebook later denied this, only saying some physical barriers had to be worked around.

Facebook says their services are now back online and they are actively working to fully return them to regular operations. They also denied that user data was compromised as a result of the downtime and apologized for the impact of the outage.