Thanks Jim. [Taking off Chair Hat] The group thus far has agreed to the premise that the evaluation for Pre-approval should be the same as what would be done in the regular evaluation after application submission. That being the case, if we look at past performance for pre-approval, then we would also have to look at it for the regular evaluation. The problem is that applying for a new TLD looks at the future performance, not past performance. If you start looking at past performance, then you have to figure out severity levels of violations. Which violations can/should be used against an RO. And does that mean that an RSP that had a violation in the past could never be eligible for being a back end operator for new TLDs. I believe the number one rule is that we need to be consistent. Historically all evaluations have been forward looking. Do you promise to abide by the contractual requirements or not and do you have the knowledge, expertise (as shown through the application) to run a registry. If yes, you pass. If no, you don’t. Then you are held to a contract. Pre-approval does not change anything. It is just saying that you have passed the evaluation before applications are submitted. Nothing more, nothing less. Jeff Neuman Senior Vice President Com Laude | Valideus D: +1.703.635.7514 E: jeff.neuman@comlaude.com<mailto:jeff.neuman@comlaude.com> From: Gnso-newgtld-wg <gnso-newgtld-wg-bounces@icann.org> On Behalf Of Jim Prendergast Sent: Monday, September 16, 2019 10:07 AM To: gnso-newgtld-wg@icann.org Subject: Re: [Gnso-newgtld-wg] SLA Monitoring (SLAM) Statistics Following up on the conversation held at the end of the last call. Jeff raised some very good points as to why having this data is important. GDD’s position that “its hard to get” just doesn’t make sense. 1) GDD clearly has this data as they created the charts in the slides. Noting that some of the charts have issues, having the raw data behind those would yield some better insight. 2) its in ICANN’s benefit to explain this with some context as opposed to throwing out raw number without any narrative. My continued concern is that it’s clear that some existing registries (and by extension their back end service providers) still are having ongoing issues with hitting SLAs. And clearly there was a major event in the not too distant past that we have no context about. I simply don’t know how ICANN can designate someone as “pre-approved” if these operational challenges are evident and ongoing. From: Jim Prendergast Sent: Friday, September 06, 2019 11:01 AM To: gnso-newgtld-wg@icann.org<mailto:gnso-newgtld-wg@icann.org> Subject: FW: SLA Monitoring (SLAM) Statistics It appears that my previous message was never circulated via the list as I had an incorrect email. This is take 2. While the topic did not come up on the aforementioned call, it is still important information for the WG to have as it considers other parts of the program including pre-approval programs. thanks From: Jim Prendergast Sent: Thursday, August 29, 2019 5:34 PM To: Austin, Donna <Donna.Austin@team.neustar<mailto:Donna.Austin@team.neustar>>; Steve Chan <steve.chan@icann.org<mailto:steve.chan@icann.org>>; Gnso-newgtld-wg <gnso-newgtld-wg-bounces@icann.org<mailto:gnso-newgtld-wg-bounces@icann.org>> Subject: RE: SLA Monitoring (SLAM) Statistics As just referenced on the call (in my best impression of Mickey Mouse voice) see my note below as well as the additions here as I’ve had more time to look at the slides. We really need the raw data and some narrative. If this will be covered on the call on Tuesday – I move that we table that discussion until we have this information and have had a chance to review and digest. Sending it out late tomorrow before a holiday weekend for many of the regular participants simply would not provide enough prep time. As I looked at the slide more a few thoughts * Even the areas that are outside the ‘significant spikes’ are interesting as they represent non-trivial levels of failure. * The DNS had 10+ failures in 23 of 58 quarters (~40%). And scale on the RDDS graph is such that the lowest line is 200 failures per quarter and the line is above that for all but two quarters over nearly 5 years. * And, of course, all of this data is less informative because it’s an aggregated industry view. The reader can’t tell which RSP is involved for any of these failures. If I was shopping for an ICANN “pre-approved” vendor – I would certainly want access to that data to ensure I factored that into my decision making * Slide 2 is missing May 2019, which could or could not be significant, given the performance of June and July. Thanks From: Jim Prendergast Sent: Monday, August 26, 2019 11:19 PM To: Austin, Donna <Donna.Austin@team.neustar<mailto:Donna.Austin@team.neustar>>; Steve Chan <steve.chan@icann.org<mailto:steve.chan@icann.org>>; Gnso-newgtld-wg <gnso-newgtld-wg-bounces@icann.org<mailto:gnso-newgtld-wg-bounces@icann.org>> Subject: RE: SLA Monitoring (SLAM) Statistics Yes. Add my thanks for this as well. And I agree with Donna on the need for more detail. I think having the raw data behind the slides would be beneficial as when I look at the chart on slide 4, the timeline seems to be out of sorts, especially as we get to 2019. It would also be useful to know if there is any explanation behind the spikes in DNS failures earlier this year and RDDS in 2018. The DNS failures are really something else. More in June and July than in the entire cumulative history of the program. Thanks From: Gnso-newgtld-wg <gnso-newgtld-wg-bounces@icann.org<mailto:gnso-newgtld-wg-bounces@icann.org>> On Behalf Of Austin, Donna via Gnso-newgtld-wg Sent: Monday, August 26, 2019 7:16 PM To: Steve Chan <steve.chan@icann.org<mailto:steve.chan@icann.org>>; gnso-newgtld-wg@icann.org<mailto:gnso-newgtld-wg@icann.org> Subject: Re: [Gnso-newgtld-wg] SLA Monitoring (SLAM) Statistics Steve, thanks for providing this update. The information provided is not as ‘complete’ as that provided to the group in 2017. I’ve attached a document from an RySG Discussion Group that I believe was provided to this group early last year that summarises the level of detail that was available at that time—I think more detail would be helpful for the purposes of this group. Generalisations are not helpful either— It’s also disappointing that only ‘anecdotal’ information is available about PDT, which I personally don’t find particularly helpful to our discussion because the source doesn’t include any insight from those that were subject to PDT. For this information to be meaningful, more detailed statistics and analysis is necessary. Donna From: Gnso-newgtld-wg [mailto:gnso-newgtld-wg-bounces@icann.org] On Behalf Of Steve Chan Sent: Friday, August 23, 2019 2:26 PM To: gnso-newgtld-wg@icann.org<mailto:gnso-newgtld-wg@icann.org> Subject: [Gnso-newgtld-wg] SLA Monitoring (SLAM) Statistics Dear WG Members, During the course of discussions, particularly as it relates to RSP pre-approval and most recently, registrant protections, WG members have asked for updated statistics on the Service Level Agreement (SLA) Monitoring program within ICANN. Attached, please find the latest report. A few contextual notes that may help your digestion of this information: * The statistics are on a per TLD basis for each month. * In some months, there are spikes in the number of DNS and RDDS failures. Usually, these spikes are a caused by a failure in an RSP that serves a large number of TLDs. * On the 4ht slide, you will see several failures that reached EBERO thresholds but in the vast majority of cases, the failures did not trigger an EBERO event. As has been shared previously, this is likely because ICANN Org worked with the registry to resolve the event and/or there were minimal domains under management. In addition, members asked for information around Pre-Delegation Testing (PDT), which I believe is now an element of the broader Registry System Testing (RST). Please see the below response from GDD: “Additionally, on the PDT/RST side - ICANN org has neither compiled nor published summary data regarding Pre-Delegation Test results from the 2012 round. Testing data was tracked and managed by IIS, the pre-delegation testing vendor. Such testing data, was not provided to ICANN org as part of the service delivery efforts. However, based on ICANN org management of the Pre-Delegation testing processing since 2012, we can share anecdotal information based on the collective recollection of org and vendor staff. These anecdotal results includes: 1. Most TLDs received follow-up questions regarding their PDT testing submission (similar to initial evaluation Clarifying Questions) 2. Approximately one quarter of TLDs tested required extended testing beyond the standard 2 to 3 week testing cycle. Some of these tests extended up to 12 weeks. 3. Based on the flexibility of extended testing and the assistance provided by the PDT vendor to TLD operators, fewer than 10 TLDs required a second full PDT testing cycle.“ Hopefully you all find this information helpful. If you have questions or concerns, especially for GDD, please let us know. Best, Steve Steven Chan
 Policy Director, GNSO Support Internet Corporation for Assigned Names and Numbers (ICANN) 12025 Waterfront Drive, Suite 300 Los Angeles, CA 90094-2536
 Email: steve.chan@icann.org<mailto:steve.chan@icann.org> Skype: steve.chan55 Mobile: +1.310.339.4410 Find out more about the GNSO by visiting: https://learn.icann.org/<https://urldefense.proofpoint.com/v2/url?u=https-3A__learn.icann.org_&d=DwMGaQ&c=FmY1u3PJp6wrcrwll3mSVzgfkbPSS6sJms7xcl4I5cM&r=DRa2dXAvSFpCIgmkXhFzL7ar9Qfqa0AIgn-H4xR2EBk&m=jLNFXvpu9gNdUeHi-G6sjWNCF9w4_AwhzzUDFZy2elE&s=o7Auz997kA-HPv9PHJCjFVZw7Pgo8krw4MxfqCwBrIU&e=> Follow @GNSO on Twitter: https://twitter.com/ICANN_GNSO<https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_ICANN-5FGNSO&d=DwMGaQ&c=FmY1u3PJp6wrcrwll3mSVzgfkbPSS6sJms7xcl4I5cM&r=DRa2dXAvSFpCIgmkXhFzL7ar9Qfqa0AIgn-H4xR2EBk&m=jLNFXvpu9gNdUeHi-G6sjWNCF9w4_AwhzzUDFZy2elE&s=kWw4fQPNjw2lVKy1UjTxS2F0BmjEAzaDFWNmsYywbmE&e=> Transcripts and recordings of GNSO Working Group and Council events are located on the GNSO Master Calendar <https://urldefense.proofpoint.com/v2/url?u=https-3A__gnso.icann.org_en_group...> See All SO and AC events on the ICANN Global calendar [features.icann.org]<https://urldefense.proofpoint.com/v2/url?u=https-3A__features.icann.org_cale...> ________________________________ The contents of this email and any attachments are confidential to the intended recipient. They may not be disclosed, used by or copied in any way by anyone other than the intended recipient. If you have received this message in error, please return it to the sender (deleting the body of the email and attachments in your reply) and immediately and permanently delete it. Please note that the Com Laude Group does not accept any responsibility for viruses and it is your responsibility to scan or otherwise check this email and any attachments. The Com Laude Group does not accept liability for statements which are clearly the sender's own and not made on behalf of the group or one of its member entities. The Com Laude Group includes Nom-IQ Limited t/a Com Laude, a company registered in England and Wales with company number 5047655 and registered office at 28-30 Little Russell Street, London, WC1A 2HN England; Valideus Limited, a company registered in England and Wales with company number 06181291 and registered office at 28-30 Little Russell Street, London, WC1A 2HN England; Demys Limited, a company registered in Scotland with company number SC197176, having its registered office at 33 Melville Street, Edinburgh, Lothian, EH3 7JF Scotland; Consonum, Inc. dba Com Laude USA and Valideus USA, headquartered at 1751 Pinnacle Drive, Suite 600, McLean, VA 22102, USA; Com Laude (Japan) Corporation, a company registered in Japan having its registered office at Suite 319,1-3-21 Shinkawa, Chuo-ku, Tokyo, 104-0033, Japan. For further information see www.comlaude.com<https://comlaude.com>

Hi Jim, This seems to be an issue with terminology. What we’ve been calling “RSP pre-approval” simply means that the process of evaluating the technical section of the gTLD application is done once per RSP, rather than inefficiently doing it multiple times for each TLD they represent. To make this more practical for applicants who might want to use a given RSP, the process can be done prior to the application window. So it’s really just shifting the timing of a process (evaluation) a few months earlier. The question of whether the SLAs set out in the application and RA are adequate is a different question, which I don’t think affects the viability of “RSP pre-approval”. If it is deemed that there should be more stringent SLAs then they will apply to all RSPs, irrespective of whether they’ve been pre-approved (evaluated) or approved (evaluated) at some other time. Since this is about evaluation, it occurs to me that perhaps “RSP pre-approval” is not the most appropriate term to use. I think “RSP pre-evaluation” might better describe and clarify the concept we’re talking about here. Kind regards, Ashley Ashley Roberts Senior Manager Valideus D: +44 (0) 20 7421 8264 E: ashley.roberts@valideus.com <mailto:ashley.roberts@valideus.com> From: Gnso-newgtld-wg <gnso-newgtld-wg-bounces@icann.org> On Behalf Of Jim Prendergast Sent: 20 September 2019 17:07 To: Jeff Neuman <jeff.neuman@comlaude.com>; gnso-newgtld-wg@icann.org Subject: Re: [Gnso-newgtld-wg] SLA Monitoring (SLAM) Statistics Jeff – I guess there are a few issues at play 1. We have RSPs who were approved by ICANN in the 2012 round that are not meeting SLAs. And from the charts circulated by Steve it appears that this is not a 1 off situation. It’s more systemic. This is a problem not only in the context of sub pro but I would argue beyond with compliance and possibly other departments at ICANN. 2. The charts themselves have errors in them so having the data which was used to complete the charts would provide better insight. 3. Without the details behind these we have no idea if this warrants a reexamination of the ICANN approval process to ensure it captures the short comings in meeting the current SLAs, a reexamination of whether the SLAs are at the appropriate level or whether these deficiencies were the result of scenarios like you described on a recent call where a maintenance window was submitted to ICANN but their system is not able to capture that and flags it as an SLA violation. We just don’t know so that’s why it behooves GDD to produce the document Donna is asking for. I would also note that during the GDD summit in May, GDD staff did allude to scenarios where providers had issues with passing Pre Delegation Testing on the 1st, second or 3 try that went beyond quirks like IDN tables. The point of my desire for more information is not to snare providers with one off failures but rather to ensure that providers with repeated, ongoing issues, if in fact there are any (you can’t simply tell from the charts) are not represented to new applicants as “preapproved.” I certainly don’t, and I’m sure Valideus wouldn’t want to be in the position of recommending someone as “ICANN Preapproved” when in fact there is a known history of poor performance. From: Jeff Neuman <jeff.neuman@comlaude.com<mailto:jeff.neuman@comlaude.com>> Sent: Monday, September 16, 2019 10:55 AM To: Jim Prendergast <jim@GALWAYSG.COM<mailto:jim@GALWAYSG.COM>>; gnso-newgtld-wg@icann.org<mailto:gnso-newgtld-wg@icann.org> Subject: RE: SLA Monitoring (SLAM) Statistics Thanks Jim. [Taking off Chair Hat] The group thus far has agreed to the premise that the evaluation for Pre-approval should be the same as what would be done in the regular evaluation after application submission. That being the case, if we look at past performance for pre-approval, then we would also have to look at it for the regular evaluation. The problem is that applying for a new TLD looks at the future performance, not past performance. If you start looking at past performance, then you have to figure out severity levels of violations. Which violations can/should be used against an RO. And does that mean that an RSP that had a violation in the past could never be eligible for being a back end operator for new TLDs. I believe the number one rule is that we need to be consistent. Historically all evaluations have been forward looking. Do you promise to abide by the contractual requirements or not and do you have the knowledge, expertise (as shown through the application) to run a registry. If yes, you pass. If no, you don’t. Then you are held to a contract. Pre-approval does not change anything. It is just saying that you have passed the evaluation before applications are submitted. Nothing more, nothing less. Jeff Neuman Senior Vice President Com Laude | Valideus D: +1.703.635.7514 E: jeff.neuman@comlaude.com<mailto:jeff.neuman@comlaude.com> From: Gnso-newgtld-wg <gnso-newgtld-wg-bounces@icann.org<mailto:gnso-newgtld-wg-bounces@icann.org>> On Behalf Of Jim Prendergast Sent: Monday, September 16, 2019 10:07 AM To: gnso-newgtld-wg@icann.org<mailto:gnso-newgtld-wg@icann.org> Subject: Re: [Gnso-newgtld-wg] SLA Monitoring (SLAM) Statistics Following up on the conversation held at the end of the last call. Jeff raised some very good points as to why having this data is important. GDD’s position that “its hard to get” just doesn’t make sense. 1) GDD clearly has this data as they created the charts in the slides. Noting that some of the charts have issues, having the raw data behind those would yield some better insight. 2) its in ICANN’s benefit to explain this with some context as opposed to throwing out raw number without any narrative. My continued concern is that it’s clear that some existing registries (and by extension their back end service providers) still are having ongoing issues with hitting SLAs. And clearly there was a major event in the not too distant past that we have no context about. I simply don’t know how ICANN can designate someone as “pre-approved” if these operational challenges are evident and ongoing. From: Jim Prendergast Sent: Friday, September 06, 2019 11:01 AM To: gnso-newgtld-wg@icann.org<mailto:gnso-newgtld-wg@icann.org> Subject: FW: SLA Monitoring (SLAM) Statistics It appears that my previous message was never circulated via the list as I had an incorrect email. This is take 2. While the topic did not come up on the aforementioned call, it is still important information for the WG to have as it considers other parts of the program including pre-approval programs. thanks From: Jim Prendergast Sent: Thursday, August 29, 2019 5:34 PM To: Austin, Donna <Donna.Austin@team.neustar<mailto:Donna.Austin@team.neustar>>; Steve Chan <steve.chan@icann.org<mailto:steve.chan@icann.org>>; Gnso-newgtld-wg <gnso-newgtld-wg-bounces@icann.org<mailto:gnso-newgtld-wg-bounces@icann.org>> Subject: RE: SLA Monitoring (SLAM) Statistics As just referenced on the call (in my best impression of Mickey Mouse voice) see my note below as well as the additions here as I’ve had more time to look at the slides. We really need the raw data and some narrative. If this will be covered on the call on Tuesday – I move that we table that discussion until we have this information and have had a chance to review and digest. Sending it out late tomorrow before a holiday weekend for many of the regular participants simply would not provide enough prep time. As I looked at the slide more a few thoughts * Even the areas that are outside the ‘significant spikes’ are interesting as they represent non-trivial levels of failure. * The DNS had 10+ failures in 23 of 58 quarters (~40%). And scale on the RDDS graph is such that the lowest line is 200 failures per quarter and the line is above that for all but two quarters over nearly 5 years. * And, of course, all of this data is less informative because it’s an aggregated industry view. The reader can’t tell which RSP is involved for any of these failures. If I was shopping for an ICANN “pre-approved” vendor – I would certainly want access to that data to ensure I factored that into my decision making * Slide 2 is missing May 2019, which could or could not be significant, given the performance of June and July. Thanks From: Jim Prendergast Sent: Monday, August 26, 2019 11:19 PM To: Austin, Donna <Donna.Austin@team.neustar<mailto:Donna.Austin@team.neustar>>; Steve Chan <steve.chan@icann.org<mailto:steve.chan@icann.org>>; Gnso-newgtld-wg <gnso-newgtld-wg-bounces@icann.org<mailto:gnso-newgtld-wg-bounces@icann.org>> Subject: RE: SLA Monitoring (SLAM) Statistics Yes. Add my thanks for this as well. And I agree with Donna on the need for more detail. I think having the raw data behind the slides would be beneficial as when I look at the chart on slide 4, the timeline seems to be out of sorts, especially as we get to 2019. It would also be useful to know if there is any explanation behind the spikes in DNS failures earlier this year and RDDS in 2018. The DNS failures are really something else. More in June and July than in the entire cumulative history of the program. Thanks From: Gnso-newgtld-wg <gnso-newgtld-wg-bounces@icann.org<mailto:gnso-newgtld-wg-bounces@icann.org>> On Behalf Of Austin, Donna via Gnso-newgtld-wg Sent: Monday, August 26, 2019 7:16 PM To: Steve Chan <steve.chan@icann.org<mailto:steve.chan@icann.org>>; gnso-newgtld-wg@icann.org<mailto:gnso-newgtld-wg@icann.org> Subject: Re: [Gnso-newgtld-wg] SLA Monitoring (SLAM) Statistics Steve, thanks for providing this update. The information provided is not as ‘complete’ as that provided to the group in 2017. I’ve attached a document from an RySG Discussion Group that I believe was provided to this group early last year that summarises the level of detail that was available at that time—I think more detail would be helpful for the purposes of this group. Generalisations are not helpful either— It’s also disappointing that only ‘anecdotal’ information is available about PDT, which I personally don’t find particularly helpful to our discussion because the source doesn’t include any insight from those that were subject to PDT. For this information to be meaningful, more detailed statistics and analysis is necessary. Donna From: Gnso-newgtld-wg [mailto:gnso-newgtld-wg-bounces@icann.org] On Behalf Of Steve Chan Sent: Friday, August 23, 2019 2:26 PM To: gnso-newgtld-wg@icann.org<mailto:gnso-newgtld-wg@icann.org> Subject: [Gnso-newgtld-wg] SLA Monitoring (SLAM) Statistics Dear WG Members, During the course of discussions, particularly as it relates to RSP pre-approval and most recently, registrant protections, WG members have asked for updated statistics on the Service Level Agreement (SLA) Monitoring program within ICANN. Attached, please find the latest report. A few contextual notes that may help your digestion of this information: * The statistics are on a per TLD basis for each month. * In some months, there are spikes in the number of DNS and RDDS failures. Usually, these spikes are a caused by a failure in an RSP that serves a large number of TLDs. * On the 4ht slide, you will see several failures that reached EBERO thresholds but in the vast majority of cases, the failures did not trigger an EBERO event. As has been shared previously, this is likely because ICANN Org worked with the registry to resolve the event and/or there were minimal domains under management. In addition, members asked for information around Pre-Delegation Testing (PDT), which I believe is now an element of the broader Registry System Testing (RST). Please see the below response from GDD: “Additionally, on the PDT/RST side - ICANN org has neither compiled nor published summary data regarding Pre-Delegation Test results from the 2012 round. Testing data was tracked and managed by IIS, the pre-delegation testing vendor. Such testing data, was not provided to ICANN org as part of the service delivery efforts. However, based on ICANN org management of the Pre-Delegation testing processing since 2012, we can share anecdotal information based on the collective recollection of org and vendor staff. These anecdotal results includes: 1. Most TLDs received follow-up questions regarding their PDT testing submission (similar to initial evaluation Clarifying Questions) 2. Approximately one quarter of TLDs tested required extended testing beyond the standard 2 to 3 week testing cycle. Some of these tests extended up to 12 weeks. 3. Based on the flexibility of extended testing and the assistance provided by the PDT vendor to TLD operators, fewer than 10 TLDs required a second full PDT testing cycle.“ Hopefully you all find this information helpful. If you have questions or concerns, especially for GDD, please let us know. Best, Steve Steven Chan
 Policy Director, GNSO Support Internet Corporation for Assigned Names and Numbers (ICANN) 12025 Waterfront Drive, Suite 300 Los Angeles, CA 90094-2536
 Email: steve.chan@icann.org<mailto:steve.chan@icann.org> Skype: steve.chan55 Mobile: +1.310.339.4410 Find out more about the GNSO by visiting: https://learn.icann.org/<https://urldefense.proofpoint.com/v2/url?u=https-3A__learn.icann.org_&d=DwMGaQ&c=FmY1u3PJp6wrcrwll3mSVzgfkbPSS6sJms7xcl4I5cM&r=DRa2dXAvSFpCIgmkXhFzL7ar9Qfqa0AIgn-H4xR2EBk&m=jLNFXvpu9gNdUeHi-G6sjWNCF9w4_AwhzzUDFZy2elE&s=o7Auz997kA-HPv9PHJCjFVZw7Pgo8krw4MxfqCwBrIU&e=> Follow @GNSO on Twitter: https://twitter.com/ICANN_GNSO<https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_ICANN-5FGNSO&d=DwMGaQ&c=FmY1u3PJp6wrcrwll3mSVzgfkbPSS6sJms7xcl4I5cM&r=DRa2dXAvSFpCIgmkXhFzL7ar9Qfqa0AIgn-H4xR2EBk&m=jLNFXvpu9gNdUeHi-G6sjWNCF9w4_AwhzzUDFZy2elE&s=kWw4fQPNjw2lVKy1UjTxS2F0BmjEAzaDFWNmsYywbmE&e=> Transcripts and recordings of GNSO Working Group and Council events are located on the GNSO Master Calendar <https://urldefense.proofpoint.com/v2/url?u=https-3A__gnso.icann.org_en_group...> See All SO and AC events on the ICANN Global calendar [features.icann.org]<https://urldefense.proofpoint.com/v2/url?u=https-3A__features.icann.org_cale...> ________________________________ The contents of this email and any attachments are confidential to the intended recipient. They may not be disclosed, used by or copied in any way by anyone other than the intended recipient. If you have received this message in error, please return it to the sender (deleting the body of the email and attachments in your reply) and immediately and permanently delete it. Please note that the Com Laude Group does not accept any responsibility for viruses and it is your responsibility to scan or otherwise check this email and any attachments. The Com Laude Group does not accept liability for statements which are clearly the sender's own and not made on behalf of the group or one of its member entities. The Com Laude Group includes Nom-IQ Limited t/a Com Laude, a company registered in England and Wales with company number 5047655 and registered office at 28-30 Little Russell Street, London, WC1A 2HN England; Valideus Limited, a company registered in England and Wales with company number 06181291 and registered office at 28-30 Little Russell Street, London, WC1A 2HN England; Demys Limited, a company registered in Scotland with company number SC197176, having its registered office at 33 Melville Street, Edinburgh, Lothian, EH3 7JF Scotland; Consonum, Inc. dba Com Laude USA and Valideus USA, headquartered at 1751 Pinnacle Drive, Suite 600, McLean, VA 22102, USA; Com Laude (Japan) Corporation, a company registered in Japan having its registered office at Suite 319,1-3-21 Shinkawa, Chuo-ku, Tokyo, 104-0033, Japan. For further information see www.comlaude.com<https://comlaude.com> ________________________________ The contents of this email and any attachments are confidential to the intended recipient. They may not be disclosed, used by or copied in any way by anyone other than the intended recipient. If you have received this message in error, please return it to the sender (deleting the body of the email and attachments in your reply) and immediately and permanently delete it. Please note that the Com Laude Group does not accept any responsibility for viruses and it is your responsibility to scan or otherwise check this email and any attachments. The Com Laude Group does not accept liability for statements which are clearly the sender's own and not made on behalf of the group or one of its member entities. The Com Laude Group includes Nom-IQ Limited t/a Com Laude, a company registered in England and Wales with company number 5047655 and registered office at 28-30 Little Russell Street, London, WC1A 2HN England; Valideus Limited, a company registered in England and Wales with company number 06181291 and registered office at 28-30 Little Russell Street, London, WC1A 2HN England; Demys Limited, a company registered in Scotland with company number SC197176, having its registered office at 33 Melville Street, Edinburgh, Lothian, EH3 7JF Scotland; Consonum, Inc. dba Com Laude USA and Valideus USA, headquartered at 1751 Pinnacle Drive, Suite 600, McLean, VA 22102, USA; Com Laude (Japan) Corporation, a company registered in Japan having its registered office at Suite 319,1-3-21 Shinkawa, Chuo-ku, Tokyo, 104-0033, Japan. For further information see www.comlaude.com<https://comlaude.com>

Apologies all, I know I’ve been AWOL for a couple of weeks and I’m trying to catch up. Jim, some comments inline below. From: Gnso-newgtld-wg [mailto:gnso-newgtld-wg-bounces@icann.org] On Behalf Of Jim Prendergast Sent: Saturday, September 21, 2019 2:07 AM To: Jeff Neuman <jeff.neuman@comlaude.com>; gnso-newgtld-wg@icann.org Subject: Re: [Gnso-newgtld-wg] SLA Monitoring (SLAM) Statistics Jeff – I guess there are a few issues at play 1) We have RSPs who were approved by ICANN in the 2012 round that are not meeting SLAs. And from the charts circulated by Steve it appears that this is not a 1 off situation. It’s more systemic. This is a problem not only in the context of sub pro but I would argue beyond with compliance and possibly other departments at ICANN. DA: Not to be pedantic, but we do need to be careful here. Registry Service Providers (RSP) were not approved by ICANN in 2012, the Registry Operator (RO) was based on an evaluation of their application, which included a technical component and the RO is the contracted party with ICANN. It is the RO that is not meeting the SLAs not the RSP. ICANN cannot take action against an RSP for a breach of SLAs, they can only take action against the RO. The charts themselves have errors in them so having the data which was used to complete the charts would provide better insight. 2) Without the details behind these we have no idea if this warrants a reexamination of the ICANN approval process to ensure it captures the short comings in meeting the current SLAs, a reexamination of whether the SLAs are at the appropriate level or whether these deficiencies were the result of scenarios like you described on a recent call where a maintenance window was submitted to ICANN but their system is not able to capture that and flags it as an SLA violation. We just don’t know so that’s why it behooves GDD to produce the document Donna is asking for. I would also note that during the GDD summit in May, GDD staff did allude to scenarios where providers had issues with passing Pre Delegation Testing on the 1st, second or 3 try that went beyond quirks like IDN tables. DA: I agree we need the details behind the chart of statistics provided by Francisco, similar to what was provided previously through 2017, for the reasons you suggest. The point of my desire for more information is not to snare providers with one off failures but rather to ensure that providers with repeated, ongoing issues, if in fact there are any (you can’t simply tell from the charts) are not represented to new applicants as “preapproved.” I certainly don’t, and I’m sure Valideus wouldn’t want to be in the position of recommending someone as “ICANN Preapproved” when in fact there is a known history of poor performance. From: Jeff Neuman <jeff.neuman@comlaude.com<mailto:jeff.neuman@comlaude.com>> Sent: Monday, September 16, 2019 10:55 AM To: Jim Prendergast <jim@GALWAYSG.COM<mailto:jim@GALWAYSG.COM>>; gnso-newgtld-wg@icann.org<mailto:gnso-newgtld-wg@icann.org> Subject: RE: SLA Monitoring (SLAM) Statistics Thanks Jim. [Taking off Chair Hat] The group thus far has agreed to the premise that the evaluation for Pre-approval should be the same as what would be done in the regular evaluation after application submission. That being the case, if we look at past performance for pre-approval, then we would also have to look at it for the regular evaluation. The problem is that applying for a new TLD looks at the future performance, not past performance. If you start looking at past performance, then you have to figure out severity levels of violations. Which violations can/should be used against an RO. And does that mean that an RSP that had a violation in the past could never be eligible for being a back end operator for new TLDs. I believe the number one rule is that we need to be consistent. Historically all evaluations have been forward looking. Do you promise to abide by the contractual requirements or not and do you have the knowledge, expertise (as shown through the application) to run a registry. If yes, you pass. If no, you don’t. Then you are held to a contract. Pre-approval does not change anything. It is just saying that you have passed the evaluation before applications are submitted. Nothing more, nothing less. Jeff Neuman Senior Vice President Com Laude | Valideus D: +1.703.635.7514 E: jeff.neuman@comlaude.com<mailto:jeff.neuman@comlaude.com> From: Gnso-newgtld-wg <gnso-newgtld-wg-bounces@icann.org<mailto:gnso-newgtld-wg-bounces@icann.org>> On Behalf Of Jim Prendergast Sent: Monday, September 16, 2019 10:07 AM To: gnso-newgtld-wg@icann.org<mailto:gnso-newgtld-wg@icann.org> Subject: Re: [Gnso-newgtld-wg] SLA Monitoring (SLAM) Statistics Following up on the conversation held at the end of the last call. Jeff raised some very good points as to why having this data is important. GDD’s position that “its hard to get” just doesn’t make sense. 1) GDD clearly has this data as they created the charts in the slides. Noting that some of the charts have issues, having the raw data behind those would yield some better insight. 2) its in ICANN’s benefit to explain this with some context as opposed to throwing out raw number without any narrative. My continued concern is that it’s clear that some existing registries (and by extension their back end service providers) still are having ongoing issues with hitting SLAs. And clearly there was a major event in the not too distant past that we have no context about. I simply don’t know how ICANN can designate someone as “pre-approved” if these operational challenges are evident and ongoing. From: Jim Prendergast Sent: Friday, September 06, 2019 11:01 AM To: gnso-newgtld-wg@icann.org<mailto:gnso-newgtld-wg@icann.org> Subject: FW: SLA Monitoring (SLAM) Statistics It appears that my previous message was never circulated via the list as I had an incorrect email. This is take 2. While the topic did not come up on the aforementioned call, it is still important information for the WG to have as it considers other parts of the program including pre-approval programs. thanks From: Jim Prendergast Sent: Thursday, August 29, 2019 5:34 PM To: Austin, Donna <Donna.Austin@team.neustar<mailto:Donna.Austin@team.neustar>>; Steve Chan <steve.chan@icann.org<mailto:steve.chan@icann.org>>; Gnso-newgtld-wg <gnso-newgtld-wg-bounces@icann.org<mailto:gnso-newgtld-wg-bounces@icann.org>> Subject: RE: SLA Monitoring (SLAM) Statistics As just referenced on the call (in my best impression of Mickey Mouse voice) see my note below as well as the additions here as I’ve had more time to look at the slides. We really need the raw data and some narrative. If this will be covered on the call on Tuesday – I move that we table that discussion until we have this information and have had a chance to review and digest. Sending it out late tomorrow before a holiday weekend for many of the regular participants simply would not provide enough prep time. As I looked at the slide more a few thoughts * Even the areas that are outside the ‘significant spikes’ are interesting as they represent non-trivial levels of failure. * The DNS had 10+ failures in 23 of 58 quarters (~40%). And scale on the RDDS graph is such that the lowest line is 200 failures per quarter and the line is above that for all but two quarters over nearly 5 years. * And, of course, all of this data is less informative because it’s an aggregated industry view. The reader can’t tell which RSP is involved for any of these failures. If I was shopping for an ICANN “pre-approved” vendor – I would certainly want access to that data to ensure I factored that into my decision making * Slide 2 is missing May 2019, which could or could not be significant, given the performance of June and July. Thanks From: Jim Prendergast Sent: Monday, August 26, 2019 11:19 PM To: Austin, Donna <Donna.Austin@team.neustar<mailto:Donna.Austin@team.neustar>>; Steve Chan <steve.chan@icann.org<mailto:steve.chan@icann.org>>; Gnso-newgtld-wg <gnso-newgtld-wg-bounces@icann.org<mailto:gnso-newgtld-wg-bounces@icann.org>> Subject: RE: SLA Monitoring (SLAM) Statistics Yes. Add my thanks for this as well. And I agree with Donna on the need for more detail. I think having the raw data behind the slides would be beneficial as when I look at the chart on slide 4, the timeline seems to be out of sorts, especially as we get to 2019. It would also be useful to know if there is any explanation behind the spikes in DNS failures earlier this year and RDDS in 2018. The DNS failures are really something else. More in June and July than in the entire cumulative history of the program. Thanks From: Gnso-newgtld-wg <gnso-newgtld-wg-bounces@icann.org<mailto:gnso-newgtld-wg-bounces@icann.org>> On Behalf Of Austin, Donna via Gnso-newgtld-wg Sent: Monday, August 26, 2019 7:16 PM To: Steve Chan <steve.chan@icann.org<mailto:steve.chan@icann.org>>; gnso-newgtld-wg@icann.org<mailto:gnso-newgtld-wg@icann.org> Subject: Re: [Gnso-newgtld-wg] SLA Monitoring (SLAM) Statistics Steve, thanks for providing this update. The information provided is not as ‘complete’ as that provided to the group in 2017. I’ve attached a document from an RySG Discussion Group that I believe was provided to this group early last year that summarises the level of detail that was available at that time—I think more detail would be helpful for the purposes of this group. Generalisations are not helpful either— It’s also disappointing that only ‘anecdotal’ information is available about PDT, which I personally don’t find particularly helpful to our discussion because the source doesn’t include any insight from those that were subject to PDT. For this information to be meaningful, more detailed statistics and analysis is necessary. Donna From: Gnso-newgtld-wg [mailto:gnso-newgtld-wg-bounces@icann.org] On Behalf Of Steve Chan Sent: Friday, August 23, 2019 2:26 PM To: gnso-newgtld-wg@icann.org<mailto:gnso-newgtld-wg@icann.org> Subject: [Gnso-newgtld-wg] SLA Monitoring (SLAM) Statistics Dear WG Members, During the course of discussions, particularly as it relates to RSP pre-approval and most recently, registrant protections, WG members have asked for updated statistics on the Service Level Agreement (SLA) Monitoring program within ICANN. Attached, please find the latest report. A few contextual notes that may help your digestion of this information: * The statistics are on a per TLD basis for each month. * In some months, there are spikes in the number of DNS and RDDS failures. Usually, these spikes are a caused by a failure in an RSP that serves a large number of TLDs. * On the 4ht slide, you will see several failures that reached EBERO thresholds but in the vast majority of cases, the failures did not trigger an EBERO event. As has been shared previously, this is likely because ICANN Org worked with the registry to resolve the event and/or there were minimal domains under management. In addition, members asked for information around Pre-Delegation Testing (PDT), which I believe is now an element of the broader Registry System Testing (RST). Please see the below response from GDD: “Additionally, on the PDT/RST side - ICANN org has neither compiled nor published summary data regarding Pre-Delegation Test results from the 2012 round. Testing data was tracked and managed by IIS, the pre-delegation testing vendor. Such testing data, was not provided to ICANN org as part of the service delivery efforts. However, based on ICANN org management of the Pre-Delegation testing processing since 2012, we can share anecdotal information based on the collective recollection of org and vendor staff. These anecdotal results includes: 1. Most TLDs received follow-up questions regarding their PDT testing submission (similar to initial evaluation Clarifying Questions) 2. Approximately one quarter of TLDs tested required extended testing beyond the standard 2 to 3 week testing cycle. Some of these tests extended up to 12 weeks. 3. Based on the flexibility of extended testing and the assistance provided by the PDT vendor to TLD operators, fewer than 10 TLDs required a second full PDT testing cycle.“ Hopefully you all find this information helpful. If you have questions or concerns, especially for GDD, please let us know. Best, Steve Steven Chan
 Policy Director, GNSO Support Internet Corporation for Assigned Names and Numbers (ICANN) 12025 Waterfront Drive, Suite 300 Los Angeles, CA 90094-2536
 Email: steve.chan@icann.org<mailto:steve.chan@icann.org> Skype: steve.chan55 Mobile: +1.310.339.4410 Find out more about the GNSO by visiting: https://learn.icann.org/<https://urldefense.proofpoint.com/v2/url?u=https-3A__learn.icann.org_&d=DwMGaQ&c=FmY1u3PJp6wrcrwll3mSVzgfkbPSS6sJms7xcl4I5cM&r=DRa2dXAvSFpCIgmkXhFzL7ar9Qfqa0AIgn-H4xR2EBk&m=jLNFXvpu9gNdUeHi-G6sjWNCF9w4_AwhzzUDFZy2elE&s=o7Auz997kA-HPv9PHJCjFVZw7Pgo8krw4MxfqCwBrIU&e=> Follow @GNSO on Twitter: https://twitter.com/ICANN_GNSO<https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_ICANN-5FGNSO&d=DwMGaQ&c=FmY1u3PJp6wrcrwll3mSVzgfkbPSS6sJms7xcl4I5cM&r=DRa2dXAvSFpCIgmkXhFzL7ar9Qfqa0AIgn-H4xR2EBk&m=jLNFXvpu9gNdUeHi-G6sjWNCF9w4_AwhzzUDFZy2elE&s=kWw4fQPNjw2lVKy1UjTxS2F0BmjEAzaDFWNmsYywbmE&e=> Transcripts and recordings of GNSO Working Group and Council events are located on the GNSO Master Calendar <https://urldefense.proofpoint.com/v2/url?u=https-3A__gnso.icann.org_en_group...> See All SO and AC events on the ICANN Global calendar [features.icann.org]<https://urldefense.proofpoint.com/v2/url?u=https-3A__features.icann.org_cale...> ________________________________ The contents of this email and any attachments are confidential to the intended recipient. They may not be disclosed, used by or copied in any way by anyone other than the intended recipient. If you have received this message in error, please return it to the sender (deleting the body of the email and attachments in your reply) and immediately and permanently delete it. Please note that the Com Laude Group does not accept any responsibility for viruses and it is your responsibility to scan or otherwise check this email and any attachments. The Com Laude Group does not accept liability for statements which are clearly the sender's own and not made on behalf of the group or one of its member entities. The Com Laude Group includes Nom-IQ Limited t/a Com Laude, a company registered in England and Wales with company number 5047655 and registered office at 28-30 Little Russell Street, London, WC1A 2HN England; Valideus Limited, a company registered in England and Wales with company number 06181291 and registered office at 28-30 Little Russell Street, London, WC1A 2HN England; Demys Limited, a company registered in Scotland with company number SC197176, having its registered office at 33 Melville Street, Edinburgh, Lothian, EH3 7JF Scotland; Consonum, Inc. dba Com Laude USA and Valideus USA, headquartered at 1751 Pinnacle Drive, Suite 600, McLean, VA 22102, USA; Com Laude (Japan) Corporation, a company registered in Japan having its registered office at Suite 319,1-3-21 Shinkawa, Chuo-ku, Tokyo, 104-0033, Japan. For further information see www.comlaude.com<https://urldefense.proofpoint.com/v2/url?u=https-3A__comlaude.com&d=DwMGaQ&c...>