Following up on the conversation held at the end of the last call. 

Jeff raised some very good points as to why having this data is important.

GDD’s position that “its hard to get” just doesn’t make sense.  1) GDD clearly has this data as they created the charts in the slides.  Noting that some of the charts have issues, having the raw data behind those would yield some better insight.  2) its in ICANN’s benefit to explain this with some context as opposed to throwing out raw number without any narrative.

My continued concern is that it’s clear that some existing registries (and by extension their back end service providers) still are having ongoing issues with hitting SLAs.  And clearly there was a major event in the not too distant past that we have no context about. 

I simply don’t know how ICANN can designate someone as “pre-approved” if these operational challenges are evident and ongoing. 

From: Jim Prendergast
Sent: Friday, September 06, 2019 11:01 AM
To: gnso-newgtld-wg@icann.org
Subject: FW: SLA Monitoring (SLAM) Statistics

It appears that my previous message was never circulated via the list as I had an incorrect email.  This is take 2.

While the topic did not come up on the aforementioned call, it is still important information for the WG to have as it considers other parts of the program including pre-approval programs.

thanks

From: Jim Prendergast
Sent: Thursday, August 29, 2019 5:34 PM
To: Austin, Donna <Donna.Austin@team.neustar>; Steve Chan <steve.chan@icann.org>; Gnso-newgtld-wg <gnso-newgtld-wg-bounces@icann.org>
Subject: RE: SLA Monitoring (SLAM) Statistics

As just referenced on the call (in my best impression of Mickey Mouse voice) see my note below as well as the additions here as I’ve had more time to look at the slides.

We really need the raw data and some narrative.  If this will be covered on the call on Tuesday – I move that we table that discussion until we have this information and have had a chance to review and digest.  Sending it out late tomorrow before a holiday weekend for many of the regular participants simply would not provide enough prep time.

As I looked at the slide more a few thoughts

• Even the areas that are outside the ‘significant spikes’ are interesting as they represent non-trivial levels of failure.
• The DNS had 10+ failures in 23 of 58 quarters (~40%).  And scale on the RDDS graph is such that the lowest line is 200 failures per quarter and the line is above that for all but two quarters over nearly 5 years.
• And, of course, all of this data is less informative because it’s an aggregated industry view.  The reader can’t tell which RSP is involved for any of these failures.  If I was shopping for an ICANN “pre-approved” vendor – I would certainly want access to that data to ensure I factored that into my decision making
• Slide 2 is missing May 2019, which could or could not be significant, given the performance of June and July.

Thanks

From: Jim Prendergast
Sent: Monday, August 26, 2019 11:19 PM
To: Austin, Donna <Donna.Austin@team.neustar>; Steve Chan <steve.chan@icann.org>; Gnso-newgtld-wg <gnso-newgtld-wg-bounces@icann.org>
Subject: RE: SLA Monitoring (SLAM) Statistics

Yes.  Add my thanks for this as well.  

And I agree with Donna on the need for more detail.

I think having the raw data behind the slides would be beneficial as when I look at the chart on slide 4, the timeline seems to be out of sorts, especially as we get to 2019.

It would also be useful to know if there is any explanation behind the spikes in DNS failures earlier this year and RDDS in 2018.  The DNS failures are really something else.  More in June and July than in the entire cumulative history of the program.  

Thanks

From: Gnso-newgtld-wg <gnso-newgtld-wg-bounces@icann.org> On Behalf Of Austin, Donna via Gnso-newgtld-wg
Sent: Monday, August 26, 2019 7:16 PM
To: Steve Chan <steve.chan@icann.org>; gnso-newgtld-wg@icann.org
Subject: Re: [Gnso-newgtld-wg] SLA Monitoring (SLAM) Statistics

Steve, thanks for providing this update.

The information provided is not as ‘complete’ as that provided to the group in 2017. I’ve attached a document from an RySG Discussion Group that I believe was provided to this group early last year that summarises the level of detail that was available at that time—I think more detail would be helpful for the purposes of this group. Generalisations are not helpful either—

It’s also disappointing that only ‘anecdotal’ information is available about PDT, which I personally don’t find particularly helpful to our discussion because the source doesn’t include any insight from those that were subject to PDT. For this information to be meaningful, more detailed statistics and analysis is necessary.

Donna

From: Gnso-newgtld-wg [mailto:gnso-newgtld-wg-bounces@icann.org] On Behalf Of Steve Chan
Sent: Friday, August 23, 2019 2:26 PM
To: gnso-newgtld-wg@icann.org
Subject: [Gnso-newgtld-wg] SLA Monitoring (SLAM) Statistics

Dear WG Members,

During the course of discussions, particularly as it relates to RSP pre-approval and most recently, registrant protections, WG members have asked for updated statistics on the Service Level Agreement (SLA) Monitoring program within ICANN. Attached, please find the latest report. A few contextual notes that may help your digestion of this information:

• The statistics are on a per TLD basis for each month.
• In some months, there are spikes in the number of DNS and RDDS failures. Usually, these spikes are a caused by a failure in an RSP that serves a large number of TLDs.
• On the 4ht slide, you will see several failures that reached EBERO thresholds but in the vast majority of cases, the failures did not trigger an EBERO event. As has been shared previously, this is likely because ICANN Org worked with the registry to resolve the event and/or there were minimal domains under management.

In addition, members asked for information around Pre-Delegation Testing (PDT), which I believe is now an element of the broader Registry System Testing (RST). Please see the below response from GDD:

“Additionally, on the PDT/RST side - ICANN org has neither compiled nor published summary data regarding Pre-Delegation Test results from the 2012 round.  Testing data was tracked and managed by IIS, the pre-delegation testing vendor.   Such testing data, was not provided to ICANN org as part of the service delivery efforts.  However, based on ICANN org management of the Pre-Delegation testing processing since 2012, we can share anecdotal information based on the collective recollection of org and vendor staff.  These anecdotal results includes:

1. Most TLDs received follow-up questions regarding their PDT testing submission (similar to initial evaluation Clarifying Questions)

2. Approximately one quarter of TLDs tested required extended testing beyond the standard 2 to 3 week testing cycle.  Some of these tests extended up to 12 weeks.

3. Based on the flexibility of extended testing and the assistance provided by the PDT vendor to TLD operators, fewer than 10 TLDs required a second full PDT testing cycle.“

Hopefully you all find this information helpful. If you have questions or concerns, especially for GDD, please let us know.

Best,

Steve

Steven Chan


Policy Director, GNSO Support

Internet Corporation for Assigned Names and Numbers (ICANN) 

12025 Waterfront Drive, Suite 300

Los Angeles, CA 90094-2536


Email: steve.chan@icann.org

Skype: steve.chan55

Mobile: +1.310.339.4410

Find out more about the GNSO by visiting: https://learn.icann.org/

Follow @GNSO on Twitter: https://twitter.com/ICANN_GNSO

Transcripts and recordings of GNSO Working Group and Council events are located on the GNSO Master Calendar

See All SO and AC events on the ICANN Global calendar [features.icann.org]