Facebook and Instagram begin to restore service after SEVEN HOUR global outage
Facebook shares are down by more than 6 percent from last week as a result of the outage on Monday
Facebook, Instagram and Whatsapp were back in business on Monday afternoon after a seven hour shutdown that cost CEO Mark Zuckerberg an estimated $7 billion.
Technicians had attempted a ‘manual reset’ of its servers on Monday afternoon, which appeared to have worked.
A small team of employees was sent to Facebook’s data center, in Santa Clara, California, to try a ‘manual reset’ of the company’s servers, according to an internal memo obtained by The New York Times.
Zuckerberg wrote on Facebook: ‘Facebook, Instagram, WhatsApp and Messenger are coming back online now.
‘Sorry for the disruption today — I know how much you rely on our services to stay connected with the people you care about.’
On Monday evening, Facebook tweeted: ‘To the huge community of people and businesses around the world who depend on us: we’re sorry.
‘We’ve been working hard to restore access to our apps and services and are happy to report they are coming back online now. Thank you for bearing with us.’
Facebook’s Chief Technology Officer offered his ‘sincere apologies’ as a global outage of the website, costing the company $50billion and CEO Mark Zuckerberg $7billion and counting.
A blog post from Facebook Monday said the outage was caused by ‘configuration changes on the backbone routers that coordinate network traffic between our data centers.’
The company described a ‘cascading effect” on communication between data centers, ‘bringing our services to a halt’.
The changes were believed to have started at 11.50am EST, and the outage was due to error and not a cyber attack.
It sparked global outages that are detrimental to online businesses and decimate the company’s ad-wielding power.
At 3.50pm, four hours after the outage began, CTO Mike Schroepfer tweeted: ‘*Sincere* apologies to everyone impacted by outages of Facebook powered services right now. We are experiencing networking issues and teams are working as fast as possible to debug and restore as fast as possible.’
Facebook’s Chief Technology Office Mike Schroepfer apologized on Monday afternoon as the outage entered its fifth hour
The cause of the outage is believed to be centered around Facebook’s Border Gateway Protocol.
The BGP allows for the exchange of routing information on the internet and takes people to the websites they want to access by helping them identify them on what is known as a domain naming system – a directory of websites.
Facebook’s changes included withdrawals which removed its properties from the domain naming system it operates and essentially made it impossible for anyone to connect to the sites because they could no longer be found online.
As the company scrambled to fix the issue, share prices fell by more than 5 percent, reducing the company’s total value from $967billion on Friday afternoon to $916billion by Monday’s closing bell. Zuckerberg’s stake shrank by $7billion.
Facebook, Instagram and WhatsApp have been broken for three hours after a series of changes to Facebook’s DNS – Domain Naming System – caused all three to ‘disappear’ from the internet on Monday
A New York Times reporter tweeted that staff at an unspecified Facebook office were unable to use their keycards to gain access to the premises on Monday. It’s unclear if that is related to the outage or not.
NetBlocks, which tracks internet outages and their impact, estimate the outage has already cost the global economy $160m (£117 million).
WhatsApp, which is also owned by Facebook, is also down
Experts say that even once the problem is fixed, it would be a nightmare task rebooting the system because it is so large.
The scandal-battered company’s shares had dipped by 5 percent on Monday amid the outage and after a whistleblower went public on Sunday night with claims of how it regularly places profit above morals.
CloudFare’s Chief Technology Office John Graham-Cunningham tweeted on Monday that Facebook accidentally ‘disappeared’ from the internet after making a ‘flurry’ of updates to its BGP – Border Gateway Protocol.
‘Between 15:50 UTC and 15:52 UTC Facebook and related properties disappeared from the Internet in a flurry of BGP updates,’ he said.
Facebook does not use CloudFare but it runs one of the world’s largest DNS resolvers. When sites go down because of failures in DNS systems, CloudFare tries to repair them.
Usman Muzaffar, SVP of engineering at CloudFare, explained to DailyMail.com: ‘Humans access information online through domain names, like facebook.com and DNS converts it into numbers, called an IP address, computers use.
This heat map shows where outages across the country. The service is down around the world and is expected to be costing $160million
Facebook Messenger’s outage was also reported on DownDetector at a similar time this afternoon
They’re some of the most popular social media apps around the world, but it appears that Facebook, Instagram, WhatsApp and Facebook Messenger have all crashed this afternoon. Above: The reports of Facebook outages reported on DownDetector
Twitter founder Jack Dorsey appeared to make light of Facebook’s plight this evening. Responding to a post which appeared to show how the facebook.com domain is for sale as a result of the outage, he jokingly asked: ‘How much?’
‘From what we understand of the actual issue —it is a globalized BGP configuration issue. In our experience, these usually are mistakes, not attacks.
‘Today, the directions for how to get to Facebook’s DNS server’s addresses weren’t available (and seem to still be unavailable). Without being able to contact the DNS servers, visitors trying to reach a Facebook property, like facebook.com, will not get an answer and so the page won’t load
Usman Muzaffar, SVP of engineering at CloudFare
‘Border Gateway Protocol (BGP) is the routing protocol for the Internet. Much like the post office processing mail, BGP picks the most efficient routes for delivering Internet traffic.
‘Today, the directions for how to get to Facebook’s DNS server’s addresses weren’t available (and seem to still be unavailable).
‘Without being able to contact the DNS servers, visitors trying to reach a Facebook property, like facebook.com, will not get an answer and so the page won’t load.’
Facebook’s spokesman, Andy Stone, said: ‘We’re aware that some people are having trouble accessing our apps and products.
‘We’re working to get things back to normal as quickly as possible, and we apologize for any inconvenience.’
Initially, there were reports that AT&T, Verizon and T-Mobile were all down too – however those reports stemmed from people being unable to access Facebook-run apps on their mobile devices.
Kevin King, Director of Communications at Verizon, said there was no outage with the network but that it appeared as though some users were having difficulty accessing certain apps or sites on their Verizon devices.
‘There is a known outage affecting some social media platforms, and that may cause some customers to believe they are having a problem with network connectivity. Our network is operating normally,’ a spokesman for AT&T said.
Cyber security specialist Jake Moore said a cyber attack was unlikely.
He told PA: ‘It could well be a human error or a software bug lurking in the shadows but whatever it is Facebook needs to do its best to mitigate the problem of causing more panic about this.
‘The biggest problem is fears over a cyber attack but as we saw from Fastly in the summer I would hedge my bets on that not being the case as we’re talking about one of the biggest companies in the world, but there’s always a chance.’
Adam Leon Smith, of BCS, The Chartered Institute for IT and a software testing expert, said: ‘The outage is caused by changes made to the Facebook network infrastructure. Many of the recent high-profile outages have been caused by similar network level events.
‘It is reported by unidentified Facebook sources on Reddit that the network changes have also prevented engineers from remotely connecting to resolve the issues, delaying resolution.
‘Notably, many organizations now define their physical infrastructure as code, but most do not apply the same level of testing rigor when they change that code, as they would when changing their core business logic.’
Brooke Erin Duffy, , a professor of communications at Cornell University, told The New York Times: ‘Today’s outage brought our reliance on Facebook — and its properties like WhatsApp and Instagram — into sharp relief.
‘The abruptness of today’s outage highlights the staggering level of precarity that structures our increasingly digitally mediated work economy.’
While businesses that rely on Facebook panicked, Twitter rejoiced in a surge in traffic.
‘Hello literally everyone’ was the tongue-in-cheek message from Twitter tonight as it took a lighthearted approach to its rival Facebook suffering a worldwide outage.
Social media users meanwhile flocked to Twitter to share memes about Facebook being down.
One shared a picture of Pixar superhero Mr Incredible saying ‘It’s showtime’ with the caption: ‘When most social media apps are down, Twitter be like…’
WHAT IS THE DOMAIN NAME SYSTEM AND HOW DOES IT WORK?
The Domain Name System, or DNS, is the directory of the internet.
Whenever you click on a link, send an email, open a mobile app, often one of the first things that has to happen is your device needs to look up the address of a domain.
There are two sides of the DNS network: the authoritative side, ie webpages and other content, and the resolver side, devices that are trying to access this content.
Every domain needs to have an authoritative DNS provider, servers which store DNS records. Amazon, Cloudflare and Google are among the bigger names in authoritative DNS server provision.
On the other side of the DNS system are resolvers. Every device that connects to the Internet needs a DNS resolver.
By default, these resolvers are automatically set by whatever network you’re connecting to.
So, for most Internet users, when they connect to an ISP, or a WiFi hot spot, or a mobile network, the network operator will dictate what DNS resolver to use.
The problem is that these DNS services are often slow and don’t respect your privacy.
What many Internet users don’t realise is that even if you’re visiting a website that is encrypted, indicated by the green padlock in your browser’s address bar, that doesn’t keep your DNS resolver from knowing the identity of all the sites you visit.
That means, by default, your ISP, every WiFi network you’ve connected to, and your mobile network provider have a list of every site you’ve visited while using them.