ICANN position on the Facebook outage
The problem with Facebook was self inflicted. Perhaps the simplest solution for ICANN would be a one page text with a graphic explaining that Facebook (or other large company) is not the Internet. The discussion on today's call seemed like a kind of regulatory overreach with a desire to have ICANN tell large companies how to construct their own network architecture. This really is not an ICANN issue. For those who haven't seen it yet, the CloudFlare blog post on what happened with Facebook is worth reading: https://blog.cloudflare.com/october-2021-facebook-outage/ Regards...jmcc -- ********************************************************** John McCormac * e-mail: jmcc@hosterstats.com MC2 * web: http://www.hosterstats.com/ 22 Viewmount * Domain Registrations Statistics Waterford * Domnomics - the business of domain names Ireland * https://amzn.to/2OPtEIO IE * Skype: hosterstats.com ********************************************************** -- This email has been checked for viruses by AVG. https://www.avg.com
IMHO a blog post by OCTO would suffice. It could even point at the Cloudflare explanation. Kindest regards, Olivier On 06/10/2021 17:43, John McCormac via CPWG wrote:
The problem with Facebook was self inflicted. Perhaps the simplest solution for ICANN would be a one page text with a graphic explaining that Facebook (or other large company) is not the Internet.
The discussion on today's call seemed like a kind of regulatory overreach with a desire to have ICANN tell large companies how to construct their own network architecture. This really is not an ICANN issue.
For those who haven't seen it yet, the CloudFlare blog post on what happened with Facebook is worth reading:
https://blog.cloudflare.com/october-2021-facebook-outage/
Regards...jmcc
I agree with Olivier's suggestion. CW
On 6 Oct 2021, at 18:15, Olivier MJ Crépin-Leblond via CPWG <cpwg@icann.org> wrote:
IMHO a blog post by OCTO would suffice. It could even point at the Cloudflare explanation. Kindest regards,
Olivier
On 06/10/2021 17:43, John McCormac via CPWG wrote:
The problem with Facebook was self inflicted. Perhaps the simplest solution for ICANN would be a one page text with a graphic explaining that Facebook (or other large company) is not the Internet.
The discussion on today's call seemed like a kind of regulatory overreach with a desire to have ICANN tell large companies how to construct their own network architecture. This really is not an ICANN issue.
For those who haven't seen it yet, the CloudFlare blog post on what happened with Facebook is worth reading:
https://blog.cloudflare.com/october-2021-facebook-outage/ <https://blog.cloudflare.com/october-2021-facebook-outage/>
Regards...jmcc
_______________________________________________ CPWG mailing list CPWG@icann.org https://mm.icann.org/mailman/listinfo/cpwg
_______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.
+1 John On Wed, Oct 6, 2021 at 10:43 AM John McCormac via CPWG <cpwg@icann.org> wrote:
The problem with Facebook was self inflicted. Perhaps the simplest solution for ICANN would be a one page text with a graphic explaining that Facebook (or other large company) is not the Internet.
The discussion on today's call seemed like a kind of regulatory overreach with a desire to have ICANN tell large companies how to construct their own network architecture. This really is not an ICANN issue.
For those who haven't seen it yet, the CloudFlare blog post on what happened with Facebook is worth reading:
https://blog.cloudflare.com/october-2021-facebook-outage/
Regards...jmcc -- ********************************************************** John McCormac * e-mail: jmcc@hosterstats.com MC2 * web: http://www.hosterstats.com/ 22 Viewmount * Domain Registrations Statistics Waterford * Domnomics - the business of domain names Ireland * https://amzn.to/2OPtEIO IE * Skype: hosterstats.com **********************************************************
-- This email has been checked for viruses by AVG. https://www.avg.com
_______________________________________________ CPWG mailing list CPWG@icann.org https://mm.icann.org/mailman/listinfo/cpwg
_______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.
+1 to this; it's certainly not an ICANN issue. Regards Sent from my mobile Kindly excuse brevity and typos Every word has consequences. Every silence does too! On Wed, 6 Oct 2021, 15:43 John McCormac via CPWG, <cpwg@icann.org> wrote:
The problem with Facebook was self inflicted. Perhaps the simplest solution for ICANN would be a one page text with a graphic explaining that Facebook (or other large company) is not the Internet.
The discussion on today's call seemed like a kind of regulatory overreach with a desire to have ICANN tell large companies how to construct their own network architecture. This really is not an ICANN issue.
For those who haven't seen it yet, the CloudFlare blog post on what happened with Facebook is worth reading:
https://blog.cloudflare.com/october-2021-facebook-outage/
Regards...jmcc -- ********************************************************** John McCormac * e-mail: jmcc@hosterstats.com MC2 * web: http://www.hosterstats.com/ 22 Viewmount * Domain Registrations Statistics Waterford * Domnomics - the business of domain names Ireland * https://amzn.to/2OPtEIO IE * Skype: hosterstats.com **********************************************************
-- This email has been checked for viruses by AVG. https://www.avg.com
_______________________________________________ CPWG mailing list CPWG@icann.org https://mm.icann.org/mailman/listinfo/cpwg
_______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.
+1 agree, it is not an ICANN issue - but in absence of a formal ICANN statement some Internet users might have a different impression r On 09.10.2021, at 10:25, Seun Ojedeji via CPWG <cpwg@icann.org<mailto:cpwg@icann.org>> wrote: +1 to this; it's certainly not an ICANN issue. Regards Sent from my mobile Kindly excuse brevity and typos Every word has consequences. Every silence does too! On Wed, 6 Oct 2021, 15:43 John McCormac via CPWG, <cpwg@icann.org<mailto:cpwg@icann.org>> wrote: The problem with Facebook was self inflicted. Perhaps the simplest solution for ICANN would be a one page text with a graphic explaining that Facebook (or other large company) is not the Internet. The discussion on today's call seemed like a kind of regulatory overreach with a desire to have ICANN tell large companies how to construct their own network architecture. This really is not an ICANN issue. For those who haven't seen it yet, the CloudFlare blog post on what happened with Facebook is worth reading: https://blog.cloudflare.com/october-2021-facebook-outage/ Regards...jmcc -- ********************************************************** John McCormac * e-mail: jmcc@hosterstats.com<mailto:jmcc@hosterstats.com> MC2 * web: http://www.hosterstats.com/ 22 Viewmount * Domain Registrations Statistics Waterford * Domnomics - the business of domain names Ireland * https://amzn.to/2OPtEIO IE * Skype: hosterstats.com<http://hosterstats.com/> ********************************************************** -- This email has been checked for viruses by AVG. https://www.avg.com<https://www.avg.com/> _______________________________________________ CPWG mailing list CPWG@icann.org<mailto:CPWG@icann.org> https://mm.icann.org/mailman/listinfo/cpwg _______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on. _______________________________________________ CPWG mailing list CPWG@icann.org<mailto:CPWG@icann.org> https://mm.icann.org/mailman/listinfo/cpwg _______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.
To the extent that ICANN would stick to the disclaimer line, and not share an opinion about the outage itself fine. Regards Sent from my mobile Kindly excuse brevity and typos Every word has consequences. Every silence does too! On Sat, 9 Oct 2021, 09:31 Roberto Gaetano via CPWG, <cpwg@icann.org> wrote:
+1 agree, it is not an ICANN issue - but in absence of a formal ICANN statement some Internet users might have a different impression r
On 09.10.2021, at 10:25, Seun Ojedeji via CPWG <cpwg@icann.org> wrote:
+1 to this; it's certainly not an ICANN issue.
Regards Sent from my mobile Kindly excuse brevity and typos Every word has consequences. Every silence does too!
On Wed, 6 Oct 2021, 15:43 John McCormac via CPWG, <cpwg@icann.org> wrote:
The problem with Facebook was self inflicted. Perhaps the simplest solution for ICANN would be a one page text with a graphic explaining that Facebook (or other large company) is not the Internet.
The discussion on today's call seemed like a kind of regulatory overreach with a desire to have ICANN tell large companies how to construct their own network architecture. This really is not an ICANN issue.
For those who haven't seen it yet, the CloudFlare blog post on what happened with Facebook is worth reading:
https://blog.cloudflare.com/october-2021-facebook-outage/
Regards...jmcc -- ********************************************************** John McCormac * e-mail: jmcc@hosterstats.com MC2 * web: http://www.hosterstats.com/ 22 Viewmount * Domain Registrations Statistics Waterford * Domnomics - the business of domain names Ireland * https://amzn.to/2OPtEIO IE * Skype: hosterstats.com **********************************************************
-- This email has been checked for viruses by AVG. https://www.avg.com
_______________________________________________ CPWG mailing list CPWG@icann.org https://mm.icann.org/mailman/listinfo/cpwg
_______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.
_______________________________________________ CPWG mailing list CPWG@icann.org https://mm.icann.org/mailman/listinfo/cpwg
_______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.
_______________________________________________ CPWG mailing list CPWG@icann.org https://mm.icann.org/mailman/listinfo/cpwg
_______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.
Agree with both Seun and the original post. It is NOT an ICANN issue. That said, a brief post from ICANN with the sort of simple text - the CloudFlare post is a really good example Holly
On Oct 9, 2021, at 7:30 PM, Roberto Gaetano via CPWG <cpwg@icann.org> wrote:
+1 agree, it is not an ICANN issue - but in absence of a formal ICANN statement some Internet users might have a different impression r
On 09.10.2021, at 10:25, Seun Ojedeji via CPWG <cpwg@icann.org <mailto:cpwg@icann.org>> wrote:
+1 to this; it's certainly not an ICANN issue.
Regards Sent from my mobile Kindly excuse brevity and typos Every word has consequences. Every silence does too!
On Wed, 6 Oct 2021, 15:43 John McCormac via CPWG, <cpwg@icann.org <mailto:cpwg@icann.org>> wrote: The problem with Facebook was self inflicted. Perhaps the simplest solution for ICANN would be a one page text with a graphic explaining that Facebook (or other large company) is not the Internet.
The discussion on today's call seemed like a kind of regulatory overreach with a desire to have ICANN tell large companies how to construct their own network architecture. This really is not an ICANN issue.
For those who haven't seen it yet, the CloudFlare blog post on what happened with Facebook is worth reading:
https://blog.cloudflare.com/october-2021-facebook-outage/ <https://blog.cloudflare.com/october-2021-facebook-outage/>
Regards...jmcc -- ********************************************************** John McCormac * e-mail: jmcc@hosterstats.com <mailto:jmcc@hosterstats.com> MC2 * web: http://www.hosterstats.com/ <http://www.hosterstats.com/> 22 Viewmount * Domain Registrations Statistics Waterford * Domnomics - the business of domain names Ireland * https://amzn.to/2OPtEIO <https://amzn.to/2OPtEIO> IE * Skype: hosterstats.com <http://hosterstats.com/> **********************************************************
-- This email has been checked for viruses by AVG. https://www.avg.com <https://www.avg.com/>
_______________________________________________ CPWG mailing list CPWG@icann.org <mailto:CPWG@icann.org> https://mm.icann.org/mailman/listinfo/cpwg <https://mm.icann.org/mailman/listinfo/cpwg>
_______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy <https://www.icann.org/privacy/policy>) and the website Terms of Service (https://www.icann.org/privacy/tos <https://www.icann.org/privacy/tos>). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on. _______________________________________________ CPWG mailing list CPWG@icann.org <mailto:CPWG@icann.org> https://mm.icann.org/mailman/listinfo/cpwg
_______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.
_______________________________________________ CPWG mailing list CPWG@icann.org https://mm.icann.org/mailman/listinfo/cpwg
_______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.
Sounds like consensus Jonathan Zuck Executive Director Innovators Network Foundation www.InnovatorsNetwork.org<http://www.InnovatorsNetwork.org> Main: +1 (202) 827-7594 Direct: +1 (202) 420-7483 ________________________________ From: CPWG <cpwg-bounces@icann.org> on behalf of h.raiche--- via CPWG <cpwg@icann.org> Sent: Saturday, October 9, 2021 6:37:10 AM To: Roberto Gaetano <roberto_gaetano@hotmail.com> Cc: CPWG <cpwg@icann.org> Subject: Re: [CPWG] ICANN position on the Facebook outage Agree with both Seun and the original post. It is NOT an ICANN issue. That said, a brief post from ICANN with the sort of simple text - the CloudFlare post is a really good example Holly On Oct 9, 2021, at 7:30 PM, Roberto Gaetano via CPWG <cpwg@icann.org<mailto:cpwg@icann.org>> wrote: +1 agree, it is not an ICANN issue - but in absence of a formal ICANN statement some Internet users might have a different impression r On 09.10.2021, at 10:25, Seun Ojedeji via CPWG <cpwg@icann.org<mailto:cpwg@icann.org>> wrote: +1 to this; it's certainly not an ICANN issue. Regards Sent from my mobile Kindly excuse brevity and typos Every word has consequences. Every silence does too! On Wed, 6 Oct 2021, 15:43 John McCormac via CPWG, <cpwg@icann.org<mailto:cpwg@icann.org>> wrote: The problem with Facebook was self inflicted. Perhaps the simplest solution for ICANN would be a one page text with a graphic explaining that Facebook (or other large company) is not the Internet. The discussion on today's call seemed like a kind of regulatory overreach with a desire to have ICANN tell large companies how to construct their own network architecture. This really is not an ICANN issue. For those who haven't seen it yet, the CloudFlare blog post on what happened with Facebook is worth reading: https://blog.cloudflare.com/october-2021-facebook-outage/ Regards...jmcc -- ********************************************************** John McCormac * e-mail: jmcc@hosterstats.com<mailto:jmcc@hosterstats.com> MC2 * web: http://www.hosterstats.com/ 22 Viewmount * Domain Registrations Statistics Waterford * Domnomics - the business of domain names Ireland * https://amzn.to/2OPtEIO IE * Skype: hosterstats.com<http://hosterstats.com/> ********************************************************** -- This email has been checked for viruses by AVG. https://www.avg.com<https://www.avg.com/> _______________________________________________ CPWG mailing list CPWG@icann.org<mailto:CPWG@icann.org> https://mm.icann.org/mailman/listinfo/cpwg _______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on. _______________________________________________ CPWG mailing list CPWG@icann.org<mailto:CPWG@icann.org> https://mm.icann.org/mailman/listinfo/cpwg _______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on. _______________________________________________ CPWG mailing list CPWG@icann.org<mailto:CPWG@icann.org> https://mm.icann.org/mailman/listinfo/cpwg _______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.
An organisation such as Facebook is most likely to have a thorough design of redundancy, and why did it fail? Isn't this a Security and Stability issue for ICANN to examine? How is this NOT an ICANN issue? Sivasubramanian M On Sat, Oct 9, 2021, 21:45 Jonathan Zuck via CPWG <cpwg@icann.org> wrote:
Sounds like consensus
Jonathan Zuck Executive Director Innovators Network Foundation www.InnovatorsNetwork.org Main: +1 (202) 827-7594 Direct: +1 (202) 420-7483 ------------------------------ *From:* CPWG <cpwg-bounces@icann.org> on behalf of h.raiche--- via CPWG < cpwg@icann.org> *Sent:* Saturday, October 9, 2021 6:37:10 AM *To:* Roberto Gaetano <roberto_gaetano@hotmail.com> *Cc:* CPWG <cpwg@icann.org> *Subject:* Re: [CPWG] ICANN position on the Facebook outage
Agree with both Seun and the original post. It is NOT an ICANN issue. That said, a brief post from ICANN with the sort of simple text - the CloudFlare post is a really good example
Holly
On Oct 9, 2021, at 7:30 PM, Roberto Gaetano via CPWG <cpwg@icann.org> wrote:
+1 agree, it is not an ICANN issue - but in absence of a formal ICANN statement some Internet users might have a different impression r
On 09.10.2021, at 10:25, Seun Ojedeji via CPWG <cpwg@icann.org> wrote:
+1 to this; it's certainly not an ICANN issue.
Regards Sent from my mobile Kindly excuse brevity and typos Every word has consequences. Every silence does too!
On Wed, 6 Oct 2021, 15:43 John McCormac via CPWG, <cpwg@icann.org> wrote:
The problem with Facebook was self inflicted. Perhaps the simplest solution for ICANN would be a one page text with a graphic explaining that Facebook (or other large company) is not the Internet.
The discussion on today's call seemed like a kind of regulatory overreach with a desire to have ICANN tell large companies how to construct their own network architecture. This really is not an ICANN issue.
For those who haven't seen it yet, the CloudFlare blog post on what happened with Facebook is worth reading:
https://blog.cloudflare.com/october-2021-facebook-outage/
Regards...jmcc -- ********************************************************** John McCormac * e-mail: jmcc@hosterstats.com MC2 * web: http://www.hosterstats.com/ 22 Viewmount * Domain Registrations Statistics Waterford * Domnomics - the business of domain names Ireland * https://amzn.to/2OPtEIO IE * Skype: hosterstats.com **********************************************************
-- This email has been checked for viruses by AVG. https://www.avg.com
_______________________________________________ CPWG mailing list CPWG@icann.org https://mm.icann.org/mailman/listinfo/cpwg
_______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.
_______________________________________________ CPWG mailing list CPWG@icann.org https://mm.icann.org/mailman/listinfo/cpwg
_______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.
_______________________________________________ CPWG mailing list CPWG@icann.org https://mm.icann.org/mailman/listinfo/cpwg
_______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.
_______________________________________________ CPWG mailing list CPWG@icann.org https://mm.icann.org/mailman/listinfo/cpwg
_______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.
Root Cause Analysis Summary of issue: This incident, on October 4, 2021, impacted Facebook’s backbone network. This resulted in disruption across all Facebook systems and products globally, including Workplace from Facebook. This incident was an internal issue and there were no malicious third parties or bad actors involved in causing the incident. Our investigation shows no impact to user data confidentiality or integrity. The underlying cause of the outage also impacted many internal systems, making it harder to diagnose and resolve the issue quickly. Cause of issue: This outage was triggered by the system that manages our global backbone network capacity. The backbone is the network Facebook has built to connect all our computing facilities together, which consists of tens of thousands of miles of fiber-optic cables crossing the globe and linking all our data centers. During a routine maintenance job, a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally. Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool prevented it from properly stopping the command. This change caused a complete disconnection of our server connections between our data centers and the internet. And that total loss of connection caused a second issue that made things worse. One of the jobs performed by our smaller facilities is to respond to DNS queries. Those queries are answered by our authoritative name servers that occupy well known IP addresses themselves, which in turn are advertised to the rest of the internet via another protocol called the Border Gateway Protocol (BGP). To ensure reliable operation, our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy network connection. In the recent outage the entire backbone was removed from operation, making these locations declare themselves unhealthy and withdraw those BGP advertisements. The end result was that our DNS servers became unreachable even though they were still operational. This made it impossible for the rest of the internet to find our servers. Workplace timeline: This incident related to a network outage that was experienced globally across Facebook services and included Workplace. The outage was live for around 6 hours, from approximately 16:40 - 23:30 BST. Steps to mitigate: The nature of the outage meant it was not possible to access our data centers through our normal means because the networks were down, and the total loss of DNS broke many of the internal tools we’d normally use to investigate and resolve outages like this. Our primary and out-of-band network access was down, so we sent engineers onsite to the data centers to have them debug the issue and restart the systems. But this took time, because these facilities are designed with high levels of physical and system security in mind. They’re hard to get into, and once you’re inside, the hardware and routers are designed to be difficult to modify even when you have physical access to them. So it took extra time to activate the secure access protocols needed to get people onsite and able to work on the servers. Only then could we confirm the issue and bring our backbone back online. Once our backbone network connectivity was restored across our data center regions, everything came back up with it. But the problem was not over — we knew that flipping our services back on all at once could potentially cause a new round of crashes due to a surge in traffic. Individual data centers were reporting dips in power usage in the range of tens of megawatts, and suddenly reversing such a dip in power consumption could put everything from electrical systems to caches at risk. In the end, our services came back up relatively quickly without any further systemwide failures. Prevention of recurrence: We’ve done extensive work hardening our systems to prevent unauthorized access, and ultimately it was this hardening that slowed us down as we tried to recover from an outage caused not by malicious activity, but an error of our own making. It is our belief that a tradeoff like this is worth it — greatly increased day-to-day security vs. a slower recovery from a rare event like this. However, we’ll also be looking for ways to simulate events like this moving forward to ensure better preparedness and ensuring that we take every measure to strengthen our testing, drills, and overall resilience to make sure events like this happen as rarely as possible. Regards, Raymond Mamattah Accra, Ghana On Sat, Oct 9, 2021, 7:48 PM sivasubramanian muthusamy via CPWG < cpwg@icann.org> wrote:
An organisation such as Facebook is most likely to have a thorough design of redundancy, and why did it fail? Isn't this a Security and Stability issue for ICANN to examine? How is this NOT an ICANN issue?
Sivasubramanian M
On Sat, Oct 9, 2021, 21:45 Jonathan Zuck via CPWG <cpwg@icann.org> wrote:
Sounds like consensus
Jonathan Zuck Executive Director Innovators Network Foundation www.InnovatorsNetwork.org Main: +1 (202) 827-7594 Direct: +1 (202) 420-7483 ------------------------------ *From:* CPWG <cpwg-bounces@icann.org> on behalf of h.raiche--- via CPWG < cpwg@icann.org> *Sent:* Saturday, October 9, 2021 6:37:10 AM *To:* Roberto Gaetano <roberto_gaetano@hotmail.com> *Cc:* CPWG <cpwg@icann.org> *Subject:* Re: [CPWG] ICANN position on the Facebook outage
Agree with both Seun and the original post. It is NOT an ICANN issue. That said, a brief post from ICANN with the sort of simple text - the CloudFlare post is a really good example
Holly
On Oct 9, 2021, at 7:30 PM, Roberto Gaetano via CPWG <cpwg@icann.org> wrote:
+1 agree, it is not an ICANN issue - but in absence of a formal ICANN statement some Internet users might have a different impression r
On 09.10.2021, at 10:25, Seun Ojedeji via CPWG <cpwg@icann.org> wrote:
+1 to this; it's certainly not an ICANN issue.
Regards Sent from my mobile Kindly excuse brevity and typos Every word has consequences. Every silence does too!
On Wed, 6 Oct 2021, 15:43 John McCormac via CPWG, <cpwg@icann.org> wrote:
The problem with Facebook was self inflicted. Perhaps the simplest solution for ICANN would be a one page text with a graphic explaining that Facebook (or other large company) is not the Internet.
The discussion on today's call seemed like a kind of regulatory overreach with a desire to have ICANN tell large companies how to construct their own network architecture. This really is not an ICANN issue.
For those who haven't seen it yet, the CloudFlare blog post on what happened with Facebook is worth reading:
https://blog.cloudflare.com/october-2021-facebook-outage/
Regards...jmcc -- ********************************************************** John McCormac * e-mail: jmcc@hosterstats.com MC2 * web: http://www.hosterstats.com/ 22 Viewmount * Domain Registrations Statistics Waterford * Domnomics - the business of domain names Ireland * https://amzn.to/2OPtEIO IE * Skype: hosterstats.com **********************************************************
-- This email has been checked for viruses by AVG. https://www.avg.com
_______________________________________________ CPWG mailing list CPWG@icann.org https://mm.icann.org/mailman/listinfo/cpwg
_______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.
_______________________________________________ CPWG mailing list CPWG@icann.org https://mm.icann.org/mailman/listinfo/cpwg
_______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.
_______________________________________________ CPWG mailing list CPWG@icann.org https://mm.icann.org/mailman/listinfo/cpwg
_______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.
_______________________________________________ CPWG mailing list CPWG@icann.org https://mm.icann.org/mailman/listinfo/cpwg
_______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.
_______________________________________________ CPWG mailing list CPWG@icann.org https://mm.icann.org/mailman/listinfo/cpwg
_______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.
Say rather that an organisation such as Facebook is should have a thorough design of redundancy. But, on the evidence, did not. Perhaps some redundancy designed in. But thorough, it evidently was not. That said, it isn't clear that their shortcomings are ICANN's problem. Or even ICANN's business. Any node can go down, and so no longer be reachable. It's not even particularly unusual. This was more widespread than most, but no different in essence. And no more in need of an ICANN response. Bill Jouris Sent from Yahoo Mail on Android On Sat, Oct 9, 2021 at 12:48 PM, sivasubramanian muthusamy via CPWG<cpwg@icann.org> wrote: _______________________________________________ CPWG mailing list CPWG@icann.org https://mm.icann.org/mailman/listinfo/cpwg _______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.
Dear Bill, I think we've established that how Facebook is running its networks is not ICANN's problem, but ICANN should make this clear in some form of blog or public posting so as to decline responsibility of the DNS infrastructure in this incident. You'll be surprised how many ignorant decision makers might be swayed by arguments that such incidents prove that Governments should be running the "critical resources". ICANN's business is limited to making sure that TLDs are run in a reliable way and there are requirements in ICANN's contracts with gTLD operators. But even when it comes to ccTLDS, ICANN's ability to mandate anything is severely limited, if at all impossible. One cannot repeat this enough. Kindest regards, Olivier On 10/10/2021 01:21, Bill Jouris via CPWG wrote:
Say rather that an organisation such as Facebook is should have a thorough design of redundancy. But, on the evidence, did not. Perhaps some redundancy designed in. But thorough, it evidently was not.
That said, it isn't clear that their shortcomings are ICANN's problem. Or even ICANN's business. Any node can go down, and so no longer be reachable. It's not even particularly unusual. This was more widespread than most, but no different in essence. And no more in need of an ICANN response.
Bill Jouris
Sent from Yahoo Mail on Android <https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_Andr...>
On Sat, Oct 9, 2021 at 12:48 PM, sivasubramanian muthusamy via CPWG <cpwg@icann.org> wrote: _______________________________________________
On Sun, Oct 10, 2021, 13:11 Olivier MJ Crépin-Leblond <ocl@gih.com> wrote:
Dear Bill,
I think we've established that how Facebook is running its networks is not ICANN's problem, but ICANN should make this clear in some form of blog or public posting so as to decline responsibility of the DNS infrastructure in this incident. You'll be surprised how many ignorant
decision makers might be swayed by arguments
I missed that part when I mentioned that ICANN could *examine* this issue. Now aware of possible distortions. that such incidents prove that Governments should be running the "critical
resources".
ICANN's business is limited to making sure that TLDs are run in a reliable
way and there are requirements in ICANN's contracts with gTLD operators. But even when it comes to ccTLDS, ICANN's ability to mandate anything is severely limited, if at all impossible. One cannot repeat this enough. Kindest regards,
Olivier
On 10/10/2021 01:21, Bill Jouris via CPWG wrote:
Say rather that an organisation such as Facebook is should have a thorough design of redundancy. But, on the evidence, did not. Perhaps some redundancy designed in. But thorough, it evidently was not.
That said, it isn't clear that their shortcomings are ICANN's problem. Or even ICANN's business. Any node can go down, and so no longer be reachable. It's not even particularly unusual. This was more widespread than most, but no different in essence. And no more in need of an ICANN response.
Bill Jouris
Sent from Yahoo Mail on Android <https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_Andr...>
On Sat, Oct 9, 2021 at 12:48 PM, sivasubramanian muthusamy via CPWG <cpwg@icann.org> <cpwg@icann.org> wrote: _______________________________________________
Gpod morniong: Background readiong for tonight's discussion: 20211007-numerous-lessons-we-can-learn-from-the-facebook-outage-and-its-mistakes <https://circleid.com/posts/20211007-numerous-lessons-we-can-learn-from-the-f...> Regards CW
On 11 Oct 2021, at 09:22, sivasubramanian muthusamy via CPWG <cpwg@icann.org> wrote:
On 13/10/2021 07:40, lists--- via CPWG wrote:
Gpod morniong: Background readiong for tonight's discussion:
20211007-numerous-lessons-we-can-learn-from-the-facebook-outage-and-its-mistakes <https://circleid.com/posts/20211007-numerous-lessons-we-can-learn-from-the-f...>
It still is not ICANN's place to tell businesses how to design their networks, Christopher, This still isn't really an ICANN issue. It is a Facebook issue. If ICANN wants to do something, then Olivier's (I think) suggestion of a blog post and reference to the Cloudflare article is the best solution. ICANN may regulate, in its own way, gTLDs but it does not regulate the Internet or how people and businesses connect to the Internet. Regards...jmcc -- ********************************************************** John McCormac * e-mail: jmcc@hosterstats.com MC2 * web: http://www.hosterstats.com/ 22 Viewmount * Domain Registrations Statistics Waterford * Domnomics - the business of domain names Ireland * https://amzn.to/2OPtEIO IE * Skype: hosterstats.com ********************************************************** -- This email has been checked for viruses by AVG. https://www.avg.com
Dear All, I am extremely sorry for missing the meeting tonight [13 October 2021]. I am traveling by road with very limited & unreliable connectivity on mobile. However, I located a 2012 ICANN document that was "very useful" for me. It is titled: "What ICANN Does and Doesn’t Do" I could not find any further updates myself. This may give us clarity in making the ICANN position clear. Sincerely, Gopal T V 0 9840121302 https://vidwan.inflibnet.ac.in/profile/57545 https://www.facebook.com/gopal.tadepalli PS: I will also post this on the ICANN Community Wiki Meeting Page ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Dr. T V Gopal Professor Department of Computer Science and Engineering College of Engineering Anna University Chennai - 600 025, INDIA Ph : (Off) 22351723 Extn. 3340 (Res) 24454753 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ On 2021-10-13 13:12, John McCormac via CPWG wrote:
On 13/10/2021 07:40, lists--- via CPWG wrote:
Gpod morniong: Background readiong for tonight's discussion:
20211007-numerous-lessons-we-can-learn-from-the-facebook-outage-and-its-mistakes <https://circleid.com/posts/20211007-numerous-lessons-we-can-learn-from-the-f...>
It still is not ICANN's place to tell businesses how to design their networks, Christopher, This still isn't really an ICANN issue. It is a Facebook issue.
If ICANN wants to do something, then Olivier's (I think) suggestion of a blog post and reference to the Cloudflare article is the best solution.
ICANN may regulate, in its own way, gTLDs but it does not regulate the Internet or how people and businesses connect to the Internet.
Regards...jmcc -- ********************************************************** John McCormac * e-mail: jmcc@hosterstats.com MC2 * web: http://www.hosterstats.com/ 22 Viewmount * Domain Registrations Statistics Waterford * Domnomics - the business of domain names Ireland * https://amzn.to/2OPtEIO IE * Skype: hosterstats.com **********************************************************
-- This email has been checked for viruses by AVG. https://www.avg.com
_______________________________________________ CPWG mailing list CPWG@icann.org https://mm.icann.org/mailman/listinfo/cpwg
_______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.
Dear CW, Geoff Huston has explained what (Has and what could have) happened so lucidly. Wondering if Facebook couldn't have avoided all this by separating it's command and control plane from the production plane; perhaps *diverse* redundant technical infrastructure and service could have minimised this risk; Facebook could also have studied MANRS standards and could have sought their help even for designing their internal routing flow between their own data servers; I wish that Facebook really surprises Geoff Huston in the coming weeks by sharing with the Internet community detailed technical reports of what really happened, with a view to take expert advice and help to make its own infrastructure far more robust. Transparency wouldn't bring about shame even in the slightest measure. It merely acknowledges the unforeseeable in this young technolgy that far exceeds anyone's ability to completely master. (This response is separate from my earlier comment, I am NOT implying here that this is anything more than an internal technical issue with Facebook. Having said that, I still chose to post this response because Facebook happens to be larger than a private enterprise and because what it does and does not do impacts at least a part of the whole world) Sivasubramanian M On Wed, Oct 13, 2021, 12:25 lists--- via CPWG <cpwg@icann.org> wrote:
Gpod morniong: Background readiong for tonight's discussion:
20211007-numerous-lessons-we-can-learn-from-the-facebook-outage-and-its-mistakes <https://circleid.com/posts/20211007-numerous-lessons-we-can-learn-from-the-f...>
Regards CW
On 11 Oct 2021, at 09:22, sivasubramanian muthusamy via CPWG < cpwg@icann.org> wrote:
_______________________________________________ CPWG mailing list CPWG@icann.org https://mm.icann.org/mailman/listinfo/cpwg
_______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.
participants (12)
-
Bill Jouris -
David Mackey -
gopal＠annauniv.edu -
h.raiche＠internode.on.net -
John McCormac -
Jonathan Zuck -
lists＠christopherwilkinson.eu -
Olivier MJ Crépin-Leblond -
Raymond Mamattah -
Roberto Gaetano -
Seun Ojedeji -
sivasubramanian muthusamy