FW: Names SLEs Data Review - Meeting Notes (21 July 2016 @ 21:00 UTC)
Dear members of the CWG, Please see below the meeting notes from Names SLEs Data Review meeting that took place on 21 July at 21:00 UTC. Regards, Yuko Green From: Nathalie Vergnolle Sent: Friday, July 22, 2016 2:09 PM To: dt1@icann.org Cc: Trang Nguyen <trang.nguyen@icann.org>; Yuko Green <yuko.green@icann.org>; Alireza Mohammadi <alireza.mohammadi@icann.org>; Naela Sarras <naela.sarras@icann.org> Subject: Names SLEs Data Review - Meeting Notes (21 July 2016 @ 21:00 UTC) Dear members of the Names SLEs work group, Please see below the meeting notes and chat history from yesterday's Names SLEs Data Review call. The presentation material, audio and AC room recordings are now posted at https://www.icann.org/stewardship-implementation under "Meetings & Work Sessions" section. *** Meeting Notes *** Names SLEs Data Review 21 Jul 16 @ 21:00 UTC Attendees: Ali Mohammadi Allan MacGillivray Chuck Gomes (RySG) Elaine Pruis - Donuts Jay Daley Jeff Neuman Jeffrey Eckhaus Kim Davies Mary Uduma Naela Sarras Nathalie Vergnolle Patricio Poblete Paul Kane Trang Nguyen W Murray Yuko Green Notes: Data report walk-thru: Link to the Names SLEs preliminary data report: http://mm.icann.org/pipermail/cwg-stewardship/attachments/20160715/92175b9a/... [Post-meeting note] Description of Change Requests Categories: * Category I: Routine updates impacting Root Zone File * Category II: Routine updates not impacting Root Zone File * Category III: Creating or Transferring a gTLD * Category IV: Creating or Transferring a ccTLD * Category V: Other Change Requests General notes: The purpose of the call was to initiate discussions with the DT-A. It was not intended that thresholds be formally set during this meeting. The recommendations presented in the report were developed by staff, and the data reflects the new SLE data that has been collected since early March 2016, when the corresponding changes in RZMS were deployed. All data collected is in calendar days. It was suggested to add to the report any DT-A recommendations that may have been made in previous discussions. The dashboard will make visible thresholds and actual performance, and will also highlight any missed thresholds. The following notes are specific to each measurement. Processing Performance (Submission): Support from group for proposed recommendation, which is aligned with DT-A's initial recommendation of 1min (95%). Time for lodgment of email requests: Group requested that a target be set for this measurement, even in the absence of data. Group suggested threshold of 2 days. Further discussions needed for this threshold. Time to return results for technical checks following submission of request via automated submission interface: Suggestion to align Category III and IV threshold to 50 min (95%) for both categories. Time to return results for subsequent performance of technical checks during retesting due to earlier failed tests: No comments from the group. Time to return results for performance of technical checks during Supplemental Technical Check phase: Suggestion to align Cat. III and IV to 5 min (95%). Time for authorization contacts to be asked to approve CR: No comments from the group. Time for response to be affirmed by IANA: No comments from the group. Time to complete all other validations and reviews by IANA and release request for implementation: Suggested thresholds: Cat I - 5d (90%) Cat II - 5d (90%) Cat III - 5d (90%) Cat IV - Group agreed that the percentage limit should be low. In the absence of data available, threshold could be set to what would be considered a reasonable time to perform the work. This measurement requires further discussion. There was no time to go through the rest of the measurements. It was agreed that further discussions will be carried out via the mail list and another call scheduled if necessary. Dashboard demo: A real-time dashboard has been developed and the beta was demonstrated to the group. It will be ready to be deployed by the time of the transition. In addition to presenting the data as customizable charts, the tool also provides the ability to download anonymized raw data to allow for further analysis. It was suggested to add the region or the language of the requester to the log file. ICANN acknowledge that this is a good suggestion, but will need to be implemented as an enhancement post transition. A request was made to share a beta version of the dashboard with the group. This request will be looked into by staff, as authorization from NTIA would be necessary. Chat History: Yuko Green: Hello, welcome to Names SLEs Data Review meeting! Paul Kane: Kim - can you remind me your job title Kim Davies: Director, Technical Services Paul Kane: Thanks Jay Daley: Just to clairfy - this document was shared a week ago (15th) not a couple of weeks ago Chuck Gomes (RySG): I am 1754 Yuko Green: Thank you Chuck! Yuko Green: Who may be the phone number 1108? Jeff Neuman: correct Mary Uduma: no audio Jeff Neuman: DTA asked for the data so that we could determine the specific metrics Jeff Neuman: correct Jeff Neuman: we were hoping to have 6 months of data, but 3 will have to do Mary Uduma: Audio breaking Yuko Green: @Mary, we hear everything without breaking up. May I suggest that you try calling back or re-logging back into the room? Jeffrey Eckhaus: the sound via computer / adobe connect is clear for me Yuko Green: @Jefferey, thank you for confirming Jay Daley: Patricio - can you mute your line? Yuko Green: All, please mute your phone/microphone when not speaking Elaine Pruis - Donuts: chuck and jeff mute? Jay Daley: Mary - can you mute your microphone please Jeffrey Eckhaus: that is why I was so quick to agree to the 60s Jeffrey Eckhaus: we had already liked that number as a group Patricio Poblete: I was referring to CWG- DT-A - SLE - for discussion in Istanbul - DRAFT.pdf Jeff Neuman: you can use business days Jeff Neuman: 2 business days Jeff Neuman: I agree with Jay, but not sure this is the SLA where is would matter Jeff Neuman: Category III seem awfully high Jeffrey Eckhaus: Hi All , need to drop off a little early, but will catch up on the last 20 minutes or so later today Elaine Pruis - Donuts: would the host please add the definitions of the categories to teh "notes" section Elaine Pruis - Donuts: or add a window with the info Jeff Neuman: I also thing there are ways to account for months were there are fewer transactions. Understand that assigning a percentage is difficult, but that should not preclude setting reasonable targets Jeff Neuman: Foe example, in a given month if there are 5 applicable transactions, rather than a percentage, you can say that 4 out of the 5 should meet the target Jay Daley: happy for the demo Jay Daley: I agree Jeff Chuck Gomes (RySG): I have to jump off so let me share a few comments: 1) With regard to Root Zone Maintainer thresholds, it is essential that performance definitions are the same as in the RZMA agreement; 2) performance measures should exclude planned outages; 3) as a CWG member and not a member of DT-A, I would really appreciate it if DT-A would eventually present its recommended thresholds to the CWG. Paul Kane: Compare DTA's Presnted report for SLE with IANA's proposal; No blanks, use historical data, no discrimination between gTLDs and ccTLDs, Need to enter into dialogue with DTA to refine, Seems to be double dipping, Add notes are to categories to help reader Paul Kane: Exclude NTIA's process time Jeff Neuman: Good comment Jay Jeff Neuman: Thanks Kim. I do think it was a good first draft and appreciate the walk through **************** Thanks, Nathalie. Direct line: +1-310-578-8957 Mobile: +1-310-938-1037 Skype: nathalie.vergnolle.icann Jabber: nathalie.vergnolle@jabber.icann.org<mailto:nathalie.vergnolle@jabber.icann.org>
Quick update following initial SLE Data Review. The call (open to all 177 CWG participants) was held on the 21st July, to give an initial overview of the SLE Data captured over April, May and June. The call had 15 attendees and I am pleased to report all DTA members were present. It was a constructive call which lasted 1 hour. Kim Davies, Director, Technical Services, IANA gave an introduction to the collected data captured over the 3 month period. There was a request that IANA staff produce a comparison table between the thresholds initially recommended by DTA and presented to the CWG in Istanbul March 2015 (https://community.icann.org/download/attachments/52891144/CWG-%20DT-A%20-%20...) and the IANA proposed thresholds gathered in the 3 month window. IANA agreed to do this, and there will be a follow up call/email. There was a request that the SLE thresholds proposed by IANA be transposed to the CWG approved format and tabulated as captured in the document IANA Service Level Expectations - APPROVED.pdf (https://community.icann.org/download/attachments/52891144/IANA%20Service%20L...) It was felt that it was inappropriate to have unspecified SLE performance thresholds and so it was suggested that historical data be used to populate data sets where insufficient data was available from the 3 month window. IANA proposed to have different performance expectations between gTLDs and ccTLDs and DTA members prefered that the IANA service standards should have the same thresholds for both ccTLDs and gTLDs. In some scenarios the IANA proposed thresholds seem to include "double dipping" (counting scenarios twice)... and all parties agreed to take a closer look to ensure transparency on performance monitoring. During the data capture window, (which still involved NTIA's review, which post transition will be removed yet) no allowance was made to remove the time taken for NTIA's involvement in the process in the IANA proposed SLE thresholds. As a result as this SLE will refer to the situation without NTIA's role, the request was made remove the NTIA's process time so emulate the SLE as close to the post-transition situation as possible. There was a request to include Advisory Notes to assist all readers disseminate the different Categories of transactions. The SLE specification is far from complete and DTA members agreed to work with IANA in the coming weeks to refine and populate the thresholds. We will be reporting back to the full CWG on a regular basis. We were also given a "preview" of the Dashboard requested by DTA, which conveys performance information pictorially with raw data also being available for custom monitoring .... I am confident the dashboard will be well received when it is formally launched. Best Paul Quoting Yuko Green <yuko.green@icann.org>:
Dear members of the CWG,
Please see below the meeting notes from Names SLEs Data Review meeting that took place on 21 July at 21:00 UTC.
Regards, Yuko Green
From: Nathalie Vergnolle Sent: Friday, July 22, 2016 2:09 PM To: dt1@icann.org Cc: Trang Nguyen <trang.nguyen@icann.org>; Yuko Green <yuko.green@icann.org>; Alireza Mohammadi <alireza.mohammadi@icann.org>; Naela Sarras <naela.sarras@icann.org> Subject: Names SLEs Data Review - Meeting Notes (21 July 2016 @ 21:00 UTC)
Dear members of the Names SLEs work group,
Please see below the meeting notes and chat history from yesterday's Names SLEs Data Review call.
The presentation material, audio and AC room recordings are now posted at https://www.icann.org/stewardship-implementation under "Meetings & Work Sessions" section.
*** Meeting Notes ***
Names SLEs Data Review 21 Jul 16 @ 21:00 UTC
Attendees: Ali Mohammadi
Allan MacGillivray
Chuck Gomes (RySG)
Elaine Pruis - Donuts
Jay Daley
Jeff Neuman
Jeffrey Eckhaus
Kim Davies
Mary Uduma
Naela Sarras
Nathalie Vergnolle
Patricio Poblete
Paul Kane
Trang Nguyen
W Murray
Yuko Green
Notes:
Data report walk-thru: Link to the Names SLEs preliminary data report:
http://mm.icann.org/pipermail/cwg-stewardship/attachments/20160715/92175b9a/...
[Post-meeting note] Description of Change Requests Categories:
* Category I: Routine updates impacting Root Zone File
* Category II: Routine updates not impacting Root Zone File
* Category III: Creating or Transferring a gTLD
* Category IV: Creating or Transferring a ccTLD
* Category V: Other Change Requests
General notes: The purpose of the call was to initiate discussions with the DT-A. It was not intended that thresholds be formally set during this meeting. The recommendations presented in the report were developed by staff, and the data reflects the new SLE data that has been collected since early March 2016, when the corresponding changes in RZMS were deployed. All data collected is in calendar days. It was suggested to add to the report any DT-A recommendations that may have been made in previous discussions. The dashboard will make visible thresholds and actual performance, and will also highlight any missed thresholds. The following notes are specific to each measurement.
Processing Performance (Submission): Support from group for proposed recommendation, which is aligned with DT-A's initial recommendation of 1min (95%).
Time for lodgment of email requests: Group requested that a target be set for this measurement, even in the absence of data. Group suggested threshold of 2 days. Further discussions needed for this threshold.
Time to return results for technical checks following submission of request via automated submission interface: Suggestion to align Category III and IV threshold to 50 min (95%) for both categories.
Time to return results for subsequent performance of technical checks during retesting due to earlier failed tests: No comments from the group.
Time to return results for performance of technical checks during Supplemental Technical Check phase: Suggestion to align Cat. III and IV to 5 min (95%).
Time for authorization contacts to be asked to approve CR: No comments from the group.
Time for response to be affirmed by IANA: No comments from the group.
Time to complete all other validations and reviews by IANA and release request for implementation: Suggested thresholds: Cat I - 5d (90%) Cat II - 5d (90%) Cat III - 5d (90%) Cat IV - Group agreed that the percentage limit should be low. In the absence of data available, threshold could be set to what would be considered a reasonable time to perform the work. This measurement requires further discussion.
There was no time to go through the rest of the measurements. It was agreed that further discussions will be carried out via the mail list and another call scheduled if necessary.
Dashboard demo: A real-time dashboard has been developed and the beta was demonstrated to the group. It will be ready to be deployed by the time of the transition. In addition to presenting the data as customizable charts, the tool also provides the ability to download anonymized raw data to allow for further analysis. It was suggested to add the region or the language of the requester to the log file. ICANN acknowledge that this is a good suggestion, but will need to be implemented as an enhancement post transition. A request was made to share a beta version of the dashboard with the group. This request will be looked into by staff, as authorization from NTIA would be necessary.
Chat History: Yuko Green: Hello, welcome to Names SLEs Data Review meeting!
Paul Kane: Kim - can you remind me your job title
Kim Davies: Director, Technical Services
Paul Kane: Thanks
Jay Daley: Just to clairfy - this document was shared a week ago (15th) not a couple of weeks ago
Chuck Gomes (RySG): I am 1754
Yuko Green: Thank you Chuck!
Yuko Green: Who may be the phone number 1108?
Jeff Neuman: correct
Mary Uduma: no audio
Jeff Neuman: DTA asked for the data so that we could determine the specific metrics
Jeff Neuman: correct
Jeff Neuman: we were hoping to have 6 months of data, but 3 will have to do
Mary Uduma: Audio breaking
Yuko Green: @Mary, we hear everything without breaking up. May I suggest that you try calling back or re-logging back into the room?
Jeffrey Eckhaus: the sound via computer / adobe connect is clear for me
Yuko Green: @Jefferey, thank you for confirming
Jay Daley: Patricio - can you mute your line?
Yuko Green: All, please mute your phone/microphone when not speaking
Elaine Pruis - Donuts: chuck and jeff mute?
Jay Daley: Mary - can you mute your microphone please
Jeffrey Eckhaus: that is why I was so quick to agree to the 60s
Jeffrey Eckhaus: we had already liked that number as a group
Patricio Poblete: I was referring to CWG- DT-A - SLE - for discussion in Istanbul - DRAFT.pdf
Jeff Neuman: you can use business days
Jeff Neuman: 2 business days
Jeff Neuman: I agree with Jay, but not sure this is the SLA where is would matter
Jeff Neuman: Category III seem awfully high
Jeffrey Eckhaus: Hi All , need to drop off a little early, but will catch up on the last 20 minutes or so later today
Elaine Pruis - Donuts: would the host please add the definitions of the categories to teh "notes" section
Elaine Pruis - Donuts: or add a window with the info
Jeff Neuman: I also thing there are ways to account for months were there are fewer transactions. Understand that assigning a percentage is difficult, but that should not preclude setting reasonable targets
Jeff Neuman: Foe example, in a given month if there are 5 applicable transactions, rather than a percentage, you can say that 4 out of the 5 should meet the target
Jay Daley: happy for the demo
Jay Daley: I agree Jeff
Chuck Gomes (RySG): I have to jump off so let me share a few comments: 1) With regard to Root Zone Maintainer thresholds, it is essential that performance definitions are the same as in the RZMA agreement; 2) performance measures should exclude planned outages; 3) as a CWG member and not a member of DT-A, I would really appreciate it if DT-A would eventually present its recommended thresholds to the CWG.
Paul Kane: Compare DTA's Presnted report for SLE with IANA's proposal; No blanks, use historical data, no discrimination between gTLDs and ccTLDs, Need to enter into dialogue with DTA to refine, Seems to be double dipping, Add notes are to categories to help reader
Paul Kane: Exclude NTIA's process time
Jeff Neuman: Good comment Jay
Jeff Neuman: Thanks Kim. I do think it was a good first draft and appreciate the walk through
****************
Thanks, Nathalie.
Direct line: +1-310-578-8957 Mobile: +1-310-938-1037 Skype: nathalie.vergnolle.icann Jabber: nathalie.vergnolle@jabber.icann.org<mailto:nathalie.vergnolle@jabber.icann.org>
Dear members of the CWG and Names SLEs work group, As agreed during the Names SLEs call held on July 21st, we are seeking additional feedback through the mailing list with regards to the proposal that was circulated on July 15th on Names SLEs. To facilitate the review, I have reformatted the data in a table that presents the various proposals received to-date. Please note that the measurements discussed in Istanbul in March-15 were different from the final proposal of the work group. Some values may have been improperly mapped in the attached table - that would be my own wrong interpretation, and all feedbacks will be appreciated! Kindly asking for all to provide their comments preferably by next Tuesday, August 2nd. A follow-up call will be set shortly thereafter as necessary. Thank you, Nathalie. Link to the July15 proposal: http://mm.icann.org/pipermail/cwg-stewardship/attachments/20160715/92175b9a/... Link to the Istanbul March 2015 document: https://community.icann.org/download/attachments/52891144/CWG-%20DT-A%20-%20... -----Original Message----- From: dt1-bounces@icann.org [mailto:dt1-bounces@icann.org] On Behalf Of Paul M Kane - CWG Sent: Saturday, July 23, 2016 3:04 PM To: Yuko Green <yuko.green@icann.org> Cc: dt1@icann.org; cwg-stewardship@icann.org Subject: Re: [SLE Team] [CWG-Stewardship] FW: Names SLEs Data Review - Meeting Notes (21 July 2016 @ 21:00 UTC) Quick update following initial SLE Data Review. The call (open to all 177 CWG participants) was held on the 21st July, to give an initial overview of the SLE Data captured over April, May and June. The call had 15 attendees and I am pleased to report all DTA members were present. It was a constructive call which lasted 1 hour. Kim Davies, Director, Technical Services, IANA gave an introduction to the collected data captured over the 3 month period. There was a request that IANA staff produce a comparison table between the thresholds initially recommended by DTA and presented to the CWG in Istanbul March 2015 (https://community.icann.org/download/attachments/52891144/CWG-%20DT-A%20-%20...) and the IANA proposed thresholds gathered in the 3 month window. IANA agreed to do this, and there will be a follow up call/email. There was a request that the SLE thresholds proposed by IANA be transposed to the CWG approved format and tabulated as captured in the document IANA Service Level Expectations - APPROVED.pdf (https://community.icann.org/download/attachments/52891144/IANA%20Service%20L...) It was felt that it was inappropriate to have unspecified SLE performance thresholds and so it was suggested that historical data be used to populate data sets where insufficient data was available from the 3 month window. IANA proposed to have different performance expectations between gTLDs and ccTLDs and DTA members prefered that the IANA service standards should have the same thresholds for both ccTLDs and gTLDs. In some scenarios the IANA proposed thresholds seem to include "double dipping" (counting scenarios twice)... and all parties agreed to take a closer look to ensure transparency on performance monitoring. During the data capture window, (which still involved NTIA's review, which post transition will be removed yet) no allowance was made to remove the time taken for NTIA's involvement in the process in the IANA proposed SLE thresholds. As a result as this SLE will refer to the situation without NTIA's role, the request was made remove the NTIA's process time so emulate the SLE as close to the post-transition situation as possible. There was a request to include Advisory Notes to assist all readers disseminate the different Categories of transactions. The SLE specification is far from complete and DTA members agreed to work with IANA in the coming weeks to refine and populate the thresholds. We will be reporting back to the full CWG on a regular basis. We were also given a "preview" of the Dashboard requested by DTA, which conveys performance information pictorially with raw data also being available for custom monitoring .... I am confident the dashboard will be well received when it is formally launched. Best Paul Quoting Yuko Green <yuko.green@icann.org>:
Dear members of the CWG,
Please see below the meeting notes from Names SLEs Data Review meeting that took place on 21 July at 21:00 UTC.
Regards, Yuko Green
From: Nathalie Vergnolle Sent: Friday, July 22, 2016 2:09 PM To: dt1@icann.org Cc: Trang Nguyen <trang.nguyen@icann.org>; Yuko Green <yuko.green@icann.org>; Alireza Mohammadi <alireza.mohammadi@icann.org>; Naela Sarras <naela.sarras@icann.org> Subject: Names SLEs Data Review - Meeting Notes (21 July 2016 @ 21:00 UTC)
Dear members of the Names SLEs work group,
Please see below the meeting notes and chat history from yesterday's Names SLEs Data Review call.
The presentation material, audio and AC room recordings are now posted at https://www.icann.org/stewardship-implementation under "Meetings & Work Sessions" section.
*** Meeting Notes ***
Names SLEs Data Review 21 Jul 16 @ 21:00 UTC
Attendees: Ali Mohammadi
Allan MacGillivray
Chuck Gomes (RySG)
Elaine Pruis - Donuts
Jay Daley
Jeff Neuman
Jeffrey Eckhaus
Kim Davies
Mary Uduma
Naela Sarras
Nathalie Vergnolle
Patricio Poblete
Paul Kane
Trang Nguyen
W Murray
Yuko Green
Notes:
Data report walk-thru: Link to the Names SLEs preliminary data report:
http://mm.icann.org/pipermail/cwg-stewardship/attachments/20160715/92175b9a/...
[Post-meeting note] Description of Change Requests Categories:
* Category I: Routine updates impacting Root Zone File
* Category II: Routine updates not impacting Root Zone File
* Category III: Creating or Transferring a gTLD
* Category IV: Creating or Transferring a ccTLD
* Category V: Other Change Requests
General notes: The purpose of the call was to initiate discussions with the DT-A. It was not intended that thresholds be formally set during this meeting. The recommendations presented in the report were developed by staff, and the data reflects the new SLE data that has been collected since early March 2016, when the corresponding changes in RZMS were deployed. All data collected is in calendar days. It was suggested to add to the report any DT-A recommendations that may have been made in previous discussions. The dashboard will make visible thresholds and actual performance, and will also highlight any missed thresholds. The following notes are specific to each measurement.
Processing Performance (Submission): Support from group for proposed recommendation, which is aligned with DT-A's initial recommendation of 1min (95%).
Time for lodgment of email requests: Group requested that a target be set for this measurement, even in the absence of data. Group suggested threshold of 2 days. Further discussions needed for this threshold.
Time to return results for technical checks following submission of request via automated submission interface: Suggestion to align Category III and IV threshold to 50 min (95%) for both categories.
Time to return results for subsequent performance of technical checks during retesting due to earlier failed tests: No comments from the group.
Time to return results for performance of technical checks during Supplemental Technical Check phase: Suggestion to align Cat. III and IV to 5 min (95%).
Time for authorization contacts to be asked to approve CR: No comments from the group.
Time for response to be affirmed by IANA: No comments from the group.
Time to complete all other validations and reviews by IANA and release request for implementation: Suggested thresholds: Cat I - 5d (90%) Cat II - 5d (90%) Cat III - 5d (90%) Cat IV - Group agreed that the percentage limit should be low. In the absence of data available, threshold could be set to what would be considered a reasonable time to perform the work. This measurement requires further discussion.
There was no time to go through the rest of the measurements. It was agreed that further discussions will be carried out via the mail list and another call scheduled if necessary.
Dashboard demo: A real-time dashboard has been developed and the beta was demonstrated to the group. It will be ready to be deployed by the time of the transition. In addition to presenting the data as customizable charts, the tool also provides the ability to download anonymized raw data to allow for further analysis. It was suggested to add the region or the language of the requester to the log file. ICANN acknowledge that this is a good suggestion, but will need to be implemented as an enhancement post transition. A request was made to share a beta version of the dashboard with the group. This request will be looked into by staff, as authorization from NTIA would be necessary.
Chat History: Yuko Green: Hello, welcome to Names SLEs Data Review meeting!
Paul Kane: Kim - can you remind me your job title
Kim Davies: Director, Technical Services
Paul Kane: Thanks
Jay Daley: Just to clairfy - this document was shared a week ago (15th) not a couple of weeks ago
Chuck Gomes (RySG): I am 1754
Yuko Green: Thank you Chuck!
Yuko Green: Who may be the phone number 1108?
Jeff Neuman: correct
Mary Uduma: no audio
Jeff Neuman: DTA asked for the data so that we could determine the specific metrics
Jeff Neuman: correct
Jeff Neuman: we were hoping to have 6 months of data, but 3 will have to do
Mary Uduma: Audio breaking
Yuko Green: @Mary, we hear everything without breaking up. May I suggest that you try calling back or re-logging back into the room?
Jeffrey Eckhaus: the sound via computer / adobe connect is clear for me
Yuko Green: @Jefferey, thank you for confirming
Jay Daley: Patricio - can you mute your line?
Yuko Green: All, please mute your phone/microphone when not speaking
Elaine Pruis - Donuts: chuck and jeff mute?
Jay Daley: Mary - can you mute your microphone please
Jeffrey Eckhaus: that is why I was so quick to agree to the 60s
Jeffrey Eckhaus: we had already liked that number as a group
Patricio Poblete: I was referring to CWG- DT-A - SLE - for discussion in Istanbul - DRAFT.pdf
Jeff Neuman: you can use business days
Jeff Neuman: 2 business days
Jeff Neuman: I agree with Jay, but not sure this is the SLA where is would matter
Jeff Neuman: Category III seem awfully high
Jeffrey Eckhaus: Hi All , need to drop off a little early, but will catch up on the last 20 minutes or so later today
Elaine Pruis - Donuts: would the host please add the definitions of the categories to teh "notes" section
Elaine Pruis - Donuts: or add a window with the info
Jeff Neuman: I also thing there are ways to account for months were there are fewer transactions. Understand that assigning a percentage is difficult, but that should not preclude setting reasonable targets
Jeff Neuman: Foe example, in a given month if there are 5 applicable transactions, rather than a percentage, you can say that 4 out of the 5 should meet the target
Jay Daley: happy for the demo
Jay Daley: I agree Jeff
Chuck Gomes (RySG): I have to jump off so let me share a few comments: 1) With regard to Root Zone Maintainer thresholds, it is essential that performance definitions are the same as in the RZMA agreement; 2) performance measures should exclude planned outages; 3) as a CWG member and not a member of DT-A, I would really appreciate it if DT-A would eventually present its recommended thresholds to the CWG.
Paul Kane: Compare DTA's Presnted report for SLE with IANA's proposal; No blanks, use historical data, no discrimination between gTLDs and ccTLDs, Need to enter into dialogue with DTA to refine, Seems to be double dipping, Add notes are to categories to help reader
Paul Kane: Exclude NTIA's process time
Jeff Neuman: Good comment Jay
Jeff Neuman: Thanks Kim. I do think it was a good first draft and appreciate the walk through
****************
Thanks, Nathalie.
Direct line: +1-310-578-8957 Mobile: +1-310-938-1037 Skype: nathalie.vergnolle.icann Jabber: nathalie.vergnolle@jabber.icann.org<mailto:nathalie.vergnolle@jabber.i cann.org>
_______________________________________________ dt1 mailing list dt1@icann.org https://mm.icann.org/mailman/listinfo/dt1
Do the comments that were made on the call last week need to be repeated? Chuck -----Original Message----- From: cwg-stewardship-bounces@icann.org [mailto:cwg-stewardship-bounces@icann.org] On Behalf Of Nathalie Vergnolle Sent: Wednesday, July 27, 2016 7:07 PM To: dt1@icann.org; cwg-stewardship@icann.org Subject: Re: [CWG-Stewardship] [SLE Team] FW: Names SLEs Data Review - Meeting Notes (21 July 2016 @ 21:00 UTC) Dear members of the CWG and Names SLEs work group, As agreed during the Names SLEs call held on July 21st, we are seeking additional feedback through the mailing list with regards to the proposal that was circulated on July 15th on Names SLEs. To facilitate the review, I have reformatted the data in a table that presents the various proposals received to-date. Please note that the measurements discussed in Istanbul in March-15 were different from the final proposal of the work group. Some values may have been improperly mapped in the attached table - that would be my own wrong interpretation, and all feedbacks will be appreciated! Kindly asking for all to provide their comments preferably by next Tuesday, August 2nd. A follow-up call will be set shortly thereafter as necessary. Thank you, Nathalie. Link to the July15 proposal: http://mm.icann.org/pipermail/cwg-stewardship/attachments/20160715/92175b9a/... Link to the Istanbul March 2015 document: https://community.icann.org/download/attachments/52891144/CWG-%20DT-A%20-%20... -----Original Message----- From: dt1-bounces@icann.org [mailto:dt1-bounces@icann.org] On Behalf Of Paul M Kane - CWG Sent: Saturday, July 23, 2016 3:04 PM To: Yuko Green <yuko.green@icann.org> Cc: dt1@icann.org; cwg-stewardship@icann.org Subject: Re: [SLE Team] [CWG-Stewardship] FW: Names SLEs Data Review - Meeting Notes (21 July 2016 @ 21:00 UTC) Quick update following initial SLE Data Review. The call (open to all 177 CWG participants) was held on the 21st July, to give an initial overview of the SLE Data captured over April, May and June. The call had 15 attendees and I am pleased to report all DTA members were present. It was a constructive call which lasted 1 hour. Kim Davies, Director, Technical Services, IANA gave an introduction to the collected data captured over the 3 month period. There was a request that IANA staff produce a comparison table between the thresholds initially recommended by DTA and presented to the CWG in Istanbul March 2015 (https://community.icann.org/download/attachments/52891144/CWG-%20DT-A%20-%20...) and the IANA proposed thresholds gathered in the 3 month window. IANA agreed to do this, and there will be a follow up call/email. There was a request that the SLE thresholds proposed by IANA be transposed to the CWG approved format and tabulated as captured in the document IANA Service Level Expectations - APPROVED.pdf (https://community.icann.org/download/attachments/52891144/IANA%20Service%20L...) It was felt that it was inappropriate to have unspecified SLE performance thresholds and so it was suggested that historical data be used to populate data sets where insufficient data was available from the 3 month window. IANA proposed to have different performance expectations between gTLDs and ccTLDs and DTA members prefered that the IANA service standards should have the same thresholds for both ccTLDs and gTLDs. In some scenarios the IANA proposed thresholds seem to include "double dipping" (counting scenarios twice)... and all parties agreed to take a closer look to ensure transparency on performance monitoring. During the data capture window, (which still involved NTIA's review, which post transition will be removed yet) no allowance was made to remove the time taken for NTIA's involvement in the process in the IANA proposed SLE thresholds. As a result as this SLE will refer to the situation without NTIA's role, the request was made remove the NTIA's process time so emulate the SLE as close to the post-transition situation as possible. There was a request to include Advisory Notes to assist all readers disseminate the different Categories of transactions. The SLE specification is far from complete and DTA members agreed to work with IANA in the coming weeks to refine and populate the thresholds. We will be reporting back to the full CWG on a regular basis. We were also given a "preview" of the Dashboard requested by DTA, which conveys performance information pictorially with raw data also being available for custom monitoring .... I am confident the dashboard will be well received when it is formally launched. Best Paul Quoting Yuko Green <yuko.green@icann.org>:
Dear members of the CWG,
Please see below the meeting notes from Names SLEs Data Review meeting that took place on 21 July at 21:00 UTC.
Regards, Yuko Green
From: Nathalie Vergnolle Sent: Friday, July 22, 2016 2:09 PM To: dt1@icann.org Cc: Trang Nguyen <trang.nguyen@icann.org>; Yuko Green <yuko.green@icann.org>; Alireza Mohammadi <alireza.mohammadi@icann.org>; Naela Sarras <naela.sarras@icann.org> Subject: Names SLEs Data Review - Meeting Notes (21 July 2016 @ 21:00 UTC)
Dear members of the Names SLEs work group,
Please see below the meeting notes and chat history from yesterday's Names SLEs Data Review call.
The presentation material, audio and AC room recordings are now posted at https://www.icann.org/stewardship-implementation under "Meetings & Work Sessions" section.
*** Meeting Notes ***
Names SLEs Data Review 21 Jul 16 @ 21:00 UTC
Attendees: Ali Mohammadi
Allan MacGillivray
Chuck Gomes (RySG)
Elaine Pruis - Donuts
Jay Daley
Jeff Neuman
Jeffrey Eckhaus
Kim Davies
Mary Uduma
Naela Sarras
Nathalie Vergnolle
Patricio Poblete
Paul Kane
Trang Nguyen
W Murray
Yuko Green
Notes:
Data report walk-thru: Link to the Names SLEs preliminary data report:
http://mm.icann.org/pipermail/cwg-stewardship/attachments/20160715/92175b9a/...
[Post-meeting note] Description of Change Requests Categories:
* Category I: Routine updates impacting Root Zone File
* Category II: Routine updates not impacting Root Zone File
* Category III: Creating or Transferring a gTLD
* Category IV: Creating or Transferring a ccTLD
* Category V: Other Change Requests
General notes: The purpose of the call was to initiate discussions with the DT-A. It was not intended that thresholds be formally set during this meeting. The recommendations presented in the report were developed by staff, and the data reflects the new SLE data that has been collected since early March 2016, when the corresponding changes in RZMS were deployed. All data collected is in calendar days. It was suggested to add to the report any DT-A recommendations that may have been made in previous discussions. The dashboard will make visible thresholds and actual performance, and will also highlight any missed thresholds. The following notes are specific to each measurement.
Processing Performance (Submission): Support from group for proposed recommendation, which is aligned with DT-A's initial recommendation of 1min (95%).
Time for lodgment of email requests: Group requested that a target be set for this measurement, even in the absence of data. Group suggested threshold of 2 days. Further discussions needed for this threshold.
Time to return results for technical checks following submission of request via automated submission interface: Suggestion to align Category III and IV threshold to 50 min (95%) for both categories.
Time to return results for subsequent performance of technical checks during retesting due to earlier failed tests: No comments from the group.
Time to return results for performance of technical checks during Supplemental Technical Check phase: Suggestion to align Cat. III and IV to 5 min (95%).
Time for authorization contacts to be asked to approve CR: No comments from the group.
Time for response to be affirmed by IANA: No comments from the group.
Time to complete all other validations and reviews by IANA and release request for implementation: Suggested thresholds: Cat I - 5d (90%) Cat II - 5d (90%) Cat III - 5d (90%) Cat IV - Group agreed that the percentage limit should be low. In the absence of data available, threshold could be set to what would be considered a reasonable time to perform the work. This measurement requires further discussion.
There was no time to go through the rest of the measurements. It was agreed that further discussions will be carried out via the mail list and another call scheduled if necessary.
Dashboard demo: A real-time dashboard has been developed and the beta was demonstrated to the group. It will be ready to be deployed by the time of the transition. In addition to presenting the data as customizable charts, the tool also provides the ability to download anonymized raw data to allow for further analysis. It was suggested to add the region or the language of the requester to the log file. ICANN acknowledge that this is a good suggestion, but will need to be implemented as an enhancement post transition. A request was made to share a beta version of the dashboard with the group. This request will be looked into by staff, as authorization from NTIA would be necessary.
Chat History: Yuko Green: Hello, welcome to Names SLEs Data Review meeting!
Paul Kane: Kim - can you remind me your job title
Kim Davies: Director, Technical Services
Paul Kane: Thanks
Jay Daley: Just to clairfy - this document was shared a week ago (15th) not a couple of weeks ago
Chuck Gomes (RySG): I am 1754
Yuko Green: Thank you Chuck!
Yuko Green: Who may be the phone number 1108?
Jeff Neuman: correct
Mary Uduma: no audio
Jeff Neuman: DTA asked for the data so that we could determine the specific metrics
Jeff Neuman: correct
Jeff Neuman: we were hoping to have 6 months of data, but 3 will have to do
Mary Uduma: Audio breaking
Yuko Green: @Mary, we hear everything without breaking up. May I suggest that you try calling back or re-logging back into the room?
Jeffrey Eckhaus: the sound via computer / adobe connect is clear for me
Yuko Green: @Jefferey, thank you for confirming
Jay Daley: Patricio - can you mute your line?
Yuko Green: All, please mute your phone/microphone when not speaking
Elaine Pruis - Donuts: chuck and jeff mute?
Jay Daley: Mary - can you mute your microphone please
Jeffrey Eckhaus: that is why I was so quick to agree to the 60s
Jeffrey Eckhaus: we had already liked that number as a group
Patricio Poblete: I was referring to CWG- DT-A - SLE - for discussion in Istanbul - DRAFT.pdf
Jeff Neuman: you can use business days
Jeff Neuman: 2 business days
Jeff Neuman: I agree with Jay, but not sure this is the SLA where is would matter
Jeff Neuman: Category III seem awfully high
Jeffrey Eckhaus: Hi All , need to drop off a little early, but will catch up on the last 20 minutes or so later today
Elaine Pruis - Donuts: would the host please add the definitions of the categories to teh "notes" section
Elaine Pruis - Donuts: or add a window with the info
Jeff Neuman: I also thing there are ways to account for months were there are fewer transactions. Understand that assigning a percentage is difficult, but that should not preclude setting reasonable targets
Jeff Neuman: Foe example, in a given month if there are 5 applicable transactions, rather than a percentage, you can say that 4 out of the 5 should meet the target
Jay Daley: happy for the demo
Jay Daley: I agree Jeff
Chuck Gomes (RySG): I have to jump off so let me share a few comments: 1) With regard to Root Zone Maintainer thresholds, it is essential that performance definitions are the same as in the RZMA agreement; 2) performance measures should exclude planned outages; 3) as a CWG member and not a member of DT-A, I would really appreciate it if DT-A would eventually present its recommended thresholds to the CWG.
Paul Kane: Compare DTA's Presnted report for SLE with IANA's proposal; No blanks, use historical data, no discrimination between gTLDs and ccTLDs, Need to enter into dialogue with DTA to refine, Seems to be double dipping, Add notes are to categories to help reader
Paul Kane: Exclude NTIA's process time
Jeff Neuman: Good comment Jay
Jeff Neuman: Thanks Kim. I do think it was a good first draft and appreciate the walk through
****************
Thanks, Nathalie.
Direct line: +1-310-578-8957 Mobile: +1-310-938-1037 Skype: nathalie.vergnolle.icann Jabber: nathalie.vergnolle@jabber.icann.org<mailto:nathalie.vergnolle@jabber.i cann.org>
_______________________________________________ dt1 mailing list dt1@icann.org https://mm.icann.org/mailman/listinfo/dt1
Hi Chuck, The proposals of the group made on the call have been captured in column F ('DT-A proposal July 16'), plus some notes in the 'Notes' section. So, unless I missed something or incorrectly captured it, the comments made on the call shouldn't need to be repeated. Thanks, Nathalie. -----Original Message----- From: Gomes, Chuck [mailto:cgomes@verisign.com] Sent: Wednesday, July 27, 2016 5:29 PM To: Nathalie Vergnolle <nathalie.vergnolle@icann.org>; dt1@icann.org; cwg-stewardship@icann.org Subject: RE: [CWG-Stewardship] [SLE Team] FW: Names SLEs Data Review - Meeting Notes (21 July 2016 @ 21:00 UTC) Do the comments that were made on the call last week need to be repeated? Chuck -----Original Message----- From: cwg-stewardship-bounces@icann.org [mailto:cwg-stewardship-bounces@icann.org] On Behalf Of Nathalie Vergnolle Sent: Wednesday, July 27, 2016 7:07 PM To: dt1@icann.org; cwg-stewardship@icann.org Subject: Re: [CWG-Stewardship] [SLE Team] FW: Names SLEs Data Review - Meeting Notes (21 July 2016 @ 21:00 UTC) Dear members of the CWG and Names SLEs work group, As agreed during the Names SLEs call held on July 21st, we are seeking additional feedback through the mailing list with regards to the proposal that was circulated on July 15th on Names SLEs. To facilitate the review, I have reformatted the data in a table that presents the various proposals received to-date. Please note that the measurements discussed in Istanbul in March-15 were different from the final proposal of the work group. Some values may have been improperly mapped in the attached table - that would be my own wrong interpretation, and all feedbacks will be appreciated! Kindly asking for all to provide their comments preferably by next Tuesday, August 2nd. A follow-up call will be set shortly thereafter as necessary. Thank you, Nathalie. Link to the July15 proposal: http://mm.icann.org/pipermail/cwg-stewardship/attachments/20160715/92175b9a/... Link to the Istanbul March 2015 document: https://community.icann.org/download/attachments/52891144/CWG-%20DT-A%20-%20... -----Original Message----- From: dt1-bounces@icann.org [mailto:dt1-bounces@icann.org] On Behalf Of Paul M Kane - CWG Sent: Saturday, July 23, 2016 3:04 PM To: Yuko Green <yuko.green@icann.org> Cc: dt1@icann.org; cwg-stewardship@icann.org Subject: Re: [SLE Team] [CWG-Stewardship] FW: Names SLEs Data Review - Meeting Notes (21 July 2016 @ 21:00 UTC) Quick update following initial SLE Data Review. The call (open to all 177 CWG participants) was held on the 21st July, to give an initial overview of the SLE Data captured over April, May and June. The call had 15 attendees and I am pleased to report all DTA members were present. It was a constructive call which lasted 1 hour. Kim Davies, Director, Technical Services, IANA gave an introduction to the collected data captured over the 3 month period. There was a request that IANA staff produce a comparison table between the thresholds initially recommended by DTA and presented to the CWG in Istanbul March 2015 (https://community.icann.org/download/attachments/52891144/CWG-%20DT-A%20-%20...) and the IANA proposed thresholds gathered in the 3 month window. IANA agreed to do this, and there will be a follow up call/email. There was a request that the SLE thresholds proposed by IANA be transposed to the CWG approved format and tabulated as captured in the document IANA Service Level Expectations - APPROVED.pdf (https://community.icann.org/download/attachments/52891144/IANA%20Service%20L...) It was felt that it was inappropriate to have unspecified SLE performance thresholds and so it was suggested that historical data be used to populate data sets where insufficient data was available from the 3 month window. IANA proposed to have different performance expectations between gTLDs and ccTLDs and DTA members prefered that the IANA service standards should have the same thresholds for both ccTLDs and gTLDs. In some scenarios the IANA proposed thresholds seem to include "double dipping" (counting scenarios twice)... and all parties agreed to take a closer look to ensure transparency on performance monitoring. During the data capture window, (which still involved NTIA's review, which post transition will be removed yet) no allowance was made to remove the time taken for NTIA's involvement in the process in the IANA proposed SLE thresholds. As a result as this SLE will refer to the situation without NTIA's role, the request was made remove the NTIA's process time so emulate the SLE as close to the post-transition situation as possible. There was a request to include Advisory Notes to assist all readers disseminate the different Categories of transactions. The SLE specification is far from complete and DTA members agreed to work with IANA in the coming weeks to refine and populate the thresholds. We will be reporting back to the full CWG on a regular basis. We were also given a "preview" of the Dashboard requested by DTA, which conveys performance information pictorially with raw data also being available for custom monitoring .... I am confident the dashboard will be well received when it is formally launched. Best Paul Quoting Yuko Green <yuko.green@icann.org>:
Dear members of the CWG,
Please see below the meeting notes from Names SLEs Data Review meeting that took place on 21 July at 21:00 UTC.
Regards, Yuko Green
From: Nathalie Vergnolle Sent: Friday, July 22, 2016 2:09 PM To: dt1@icann.org Cc: Trang Nguyen <trang.nguyen@icann.org>; Yuko Green <yuko.green@icann.org>; Alireza Mohammadi <alireza.mohammadi@icann.org>; Naela Sarras <naela.sarras@icann.org> Subject: Names SLEs Data Review - Meeting Notes (21 July 2016 @ 21:00 UTC)
Dear members of the Names SLEs work group,
Please see below the meeting notes and chat history from yesterday's Names SLEs Data Review call.
The presentation material, audio and AC room recordings are now posted at https://www.icann.org/stewardship-implementation under "Meetings & Work Sessions" section.
*** Meeting Notes ***
Names SLEs Data Review 21 Jul 16 @ 21:00 UTC
Attendees: Ali Mohammadi
Allan MacGillivray
Chuck Gomes (RySG)
Elaine Pruis - Donuts
Jay Daley
Jeff Neuman
Jeffrey Eckhaus
Kim Davies
Mary Uduma
Naela Sarras
Nathalie Vergnolle
Patricio Poblete
Paul Kane
Trang Nguyen
W Murray
Yuko Green
Notes:
Data report walk-thru: Link to the Names SLEs preliminary data report:
http://mm.icann.org/pipermail/cwg-stewardship/attachments/20160715/92175b9a/...
[Post-meeting note] Description of Change Requests Categories:
* Category I: Routine updates impacting Root Zone File
* Category II: Routine updates not impacting Root Zone File
* Category III: Creating or Transferring a gTLD
* Category IV: Creating or Transferring a ccTLD
* Category V: Other Change Requests
General notes: The purpose of the call was to initiate discussions with the DT-A. It was not intended that thresholds be formally set during this meeting. The recommendations presented in the report were developed by staff, and the data reflects the new SLE data that has been collected since early March 2016, when the corresponding changes in RZMS were deployed. All data collected is in calendar days. It was suggested to add to the report any DT-A recommendations that may have been made in previous discussions. The dashboard will make visible thresholds and actual performance, and will also highlight any missed thresholds. The following notes are specific to each measurement.
Processing Performance (Submission): Support from group for proposed recommendation, which is aligned with DT-A's initial recommendation of 1min (95%).
Time for lodgment of email requests: Group requested that a target be set for this measurement, even in the absence of data. Group suggested threshold of 2 days. Further discussions needed for this threshold.
Time to return results for technical checks following submission of request via automated submission interface: Suggestion to align Category III and IV threshold to 50 min (95%) for both categories.
Time to return results for subsequent performance of technical checks during retesting due to earlier failed tests: No comments from the group.
Time to return results for performance of technical checks during Supplemental Technical Check phase: Suggestion to align Cat. III and IV to 5 min (95%).
Time for authorization contacts to be asked to approve CR: No comments from the group.
Time for response to be affirmed by IANA: No comments from the group.
Time to complete all other validations and reviews by IANA and release request for implementation: Suggested thresholds: Cat I - 5d (90%) Cat II - 5d (90%) Cat III - 5d (90%) Cat IV - Group agreed that the percentage limit should be low. In the absence of data available, threshold could be set to what would be considered a reasonable time to perform the work. This measurement requires further discussion.
There was no time to go through the rest of the measurements. It was agreed that further discussions will be carried out via the mail list and another call scheduled if necessary.
Dashboard demo: A real-time dashboard has been developed and the beta was demonstrated to the group. It will be ready to be deployed by the time of the transition. In addition to presenting the data as customizable charts, the tool also provides the ability to download anonymized raw data to allow for further analysis. It was suggested to add the region or the language of the requester to the log file. ICANN acknowledge that this is a good suggestion, but will need to be implemented as an enhancement post transition. A request was made to share a beta version of the dashboard with the group. This request will be looked into by staff, as authorization from NTIA would be necessary.
Chat History: Yuko Green: Hello, welcome to Names SLEs Data Review meeting!
Paul Kane: Kim - can you remind me your job title
Kim Davies: Director, Technical Services
Paul Kane: Thanks
Jay Daley: Just to clairfy - this document was shared a week ago (15th) not a couple of weeks ago
Chuck Gomes (RySG): I am 1754
Yuko Green: Thank you Chuck!
Yuko Green: Who may be the phone number 1108?
Jeff Neuman: correct
Mary Uduma: no audio
Jeff Neuman: DTA asked for the data so that we could determine the specific metrics
Jeff Neuman: correct
Jeff Neuman: we were hoping to have 6 months of data, but 3 will have to do
Mary Uduma: Audio breaking
Yuko Green: @Mary, we hear everything without breaking up. May I suggest that you try calling back or re-logging back into the room?
Jeffrey Eckhaus: the sound via computer / adobe connect is clear for me
Yuko Green: @Jefferey, thank you for confirming
Jay Daley: Patricio - can you mute your line?
Yuko Green: All, please mute your phone/microphone when not speaking
Elaine Pruis - Donuts: chuck and jeff mute?
Jay Daley: Mary - can you mute your microphone please
Jeffrey Eckhaus: that is why I was so quick to agree to the 60s
Jeffrey Eckhaus: we had already liked that number as a group
Patricio Poblete: I was referring to CWG- DT-A - SLE - for discussion in Istanbul - DRAFT.pdf
Jeff Neuman: you can use business days
Jeff Neuman: 2 business days
Jeff Neuman: I agree with Jay, but not sure this is the SLA where is would matter
Jeff Neuman: Category III seem awfully high
Jeffrey Eckhaus: Hi All , need to drop off a little early, but will catch up on the last 20 minutes or so later today
Elaine Pruis - Donuts: would the host please add the definitions of the categories to teh "notes" section
Elaine Pruis - Donuts: or add a window with the info
Jeff Neuman: I also thing there are ways to account for months were there are fewer transactions. Understand that assigning a percentage is difficult, but that should not preclude setting reasonable targets
Jeff Neuman: Foe example, in a given month if there are 5 applicable transactions, rather than a percentage, you can say that 4 out of the 5 should meet the target
Jay Daley: happy for the demo
Jay Daley: I agree Jeff
Chuck Gomes (RySG): I have to jump off so let me share a few comments: 1) With regard to Root Zone Maintainer thresholds, it is essential that performance definitions are the same as in the RZMA agreement; 2) performance measures should exclude planned outages; 3) as a CWG member and not a member of DT-A, I would really appreciate it if DT-A would eventually present its recommended thresholds to the CWG.
Paul Kane: Compare DTA's Presnted report for SLE with IANA's proposal; No blanks, use historical data, no discrimination between gTLDs and ccTLDs, Need to enter into dialogue with DTA to refine, Seems to be double dipping, Add notes are to categories to help reader
Paul Kane: Exclude NTIA's process time
Jeff Neuman: Good comment Jay
Jeff Neuman: Thanks Kim. I do think it was a good first draft and appreciate the walk through
****************
Thanks, Nathalie.
Direct line: +1-310-578-8957 Mobile: +1-310-938-1037 Skype: nathalie.vergnolle.icann Jabber: nathalie.vergnolle@jabber.icann.org<mailto:nathalie.vergnolle@jabber.i cann.org>
_______________________________________________ dt1 mailing list dt1@icann.org https://mm.icann.org/mailman/listinfo/dt1
Thanks Nathalie. Chuck -----Original Message----- From: Nathalie Vergnolle [mailto:nathalie.vergnolle@icann.org] Sent: Thursday, July 28, 2016 1:39 AM To: Gomes, Chuck; dt1@icann.org; cwg-stewardship@icann.org Subject: RE: [CWG-Stewardship] [SLE Team] FW: Names SLEs Data Review - Meeting Notes (21 July 2016 @ 21:00 UTC) Hi Chuck, The proposals of the group made on the call have been captured in column F ('DT-A proposal July 16'), plus some notes in the 'Notes' section. So, unless I missed something or incorrectly captured it, the comments made on the call shouldn't need to be repeated. Thanks, Nathalie. -----Original Message----- From: Gomes, Chuck [mailto:cgomes@verisign.com] Sent: Wednesday, July 27, 2016 5:29 PM To: Nathalie Vergnolle <nathalie.vergnolle@icann.org>; dt1@icann.org; cwg-stewardship@icann.org Subject: RE: [CWG-Stewardship] [SLE Team] FW: Names SLEs Data Review - Meeting Notes (21 July 2016 @ 21:00 UTC) Do the comments that were made on the call last week need to be repeated? Chuck -----Original Message----- From: cwg-stewardship-bounces@icann.org [mailto:cwg-stewardship-bounces@icann.org] On Behalf Of Nathalie Vergnolle Sent: Wednesday, July 27, 2016 7:07 PM To: dt1@icann.org; cwg-stewardship@icann.org Subject: Re: [CWG-Stewardship] [SLE Team] FW: Names SLEs Data Review - Meeting Notes (21 July 2016 @ 21:00 UTC) Dear members of the CWG and Names SLEs work group, As agreed during the Names SLEs call held on July 21st, we are seeking additional feedback through the mailing list with regards to the proposal that was circulated on July 15th on Names SLEs. To facilitate the review, I have reformatted the data in a table that presents the various proposals received to-date. Please note that the measurements discussed in Istanbul in March-15 were different from the final proposal of the work group. Some values may have been improperly mapped in the attached table - that would be my own wrong interpretation, and all feedbacks will be appreciated! Kindly asking for all to provide their comments preferably by next Tuesday, August 2nd. A follow-up call will be set shortly thereafter as necessary. Thank you, Nathalie. Link to the July15 proposal: http://mm.icann.org/pipermail/cwg-stewardship/attachments/20160715/92175b9a/... Link to the Istanbul March 2015 document: https://community.icann.org/download/attachments/52891144/CWG-%20DT-A%20-%20... -----Original Message----- From: dt1-bounces@icann.org [mailto:dt1-bounces@icann.org] On Behalf Of Paul M Kane - CWG Sent: Saturday, July 23, 2016 3:04 PM To: Yuko Green <yuko.green@icann.org> Cc: dt1@icann.org; cwg-stewardship@icann.org Subject: Re: [SLE Team] [CWG-Stewardship] FW: Names SLEs Data Review - Meeting Notes (21 July 2016 @ 21:00 UTC) Quick update following initial SLE Data Review. The call (open to all 177 CWG participants) was held on the 21st July, to give an initial overview of the SLE Data captured over April, May and June. The call had 15 attendees and I am pleased to report all DTA members were present. It was a constructive call which lasted 1 hour. Kim Davies, Director, Technical Services, IANA gave an introduction to the collected data captured over the 3 month period. There was a request that IANA staff produce a comparison table between the thresholds initially recommended by DTA and presented to the CWG in Istanbul March 2015 (https://community.icann.org/download/attachments/52891144/CWG-%20DT-A%20-%20...) and the IANA proposed thresholds gathered in the 3 month window. IANA agreed to do this, and there will be a follow up call/email. There was a request that the SLE thresholds proposed by IANA be transposed to the CWG approved format and tabulated as captured in the document IANA Service Level Expectations - APPROVED.pdf (https://community.icann.org/download/attachments/52891144/IANA%20Service%20L...) It was felt that it was inappropriate to have unspecified SLE performance thresholds and so it was suggested that historical data be used to populate data sets where insufficient data was available from the 3 month window. IANA proposed to have different performance expectations between gTLDs and ccTLDs and DTA members prefered that the IANA service standards should have the same thresholds for both ccTLDs and gTLDs. In some scenarios the IANA proposed thresholds seem to include "double dipping" (counting scenarios twice)... and all parties agreed to take a closer look to ensure transparency on performance monitoring. During the data capture window, (which still involved NTIA's review, which post transition will be removed yet) no allowance was made to remove the time taken for NTIA's involvement in the process in the IANA proposed SLE thresholds. As a result as this SLE will refer to the situation without NTIA's role, the request was made remove the NTIA's process time so emulate the SLE as close to the post-transition situation as possible. There was a request to include Advisory Notes to assist all readers disseminate the different Categories of transactions. The SLE specification is far from complete and DTA members agreed to work with IANA in the coming weeks to refine and populate the thresholds. We will be reporting back to the full CWG on a regular basis. We were also given a "preview" of the Dashboard requested by DTA, which conveys performance information pictorially with raw data also being available for custom monitoring .... I am confident the dashboard will be well received when it is formally launched. Best Paul Quoting Yuko Green <yuko.green@icann.org>:
Dear members of the CWG,
Please see below the meeting notes from Names SLEs Data Review meeting that took place on 21 July at 21:00 UTC.
Regards, Yuko Green
From: Nathalie Vergnolle Sent: Friday, July 22, 2016 2:09 PM To: dt1@icann.org Cc: Trang Nguyen <trang.nguyen@icann.org>; Yuko Green <yuko.green@icann.org>; Alireza Mohammadi <alireza.mohammadi@icann.org>; Naela Sarras <naela.sarras@icann.org> Subject: Names SLEs Data Review - Meeting Notes (21 July 2016 @ 21:00 UTC)
Dear members of the Names SLEs work group,
Please see below the meeting notes and chat history from yesterday's Names SLEs Data Review call.
The presentation material, audio and AC room recordings are now posted at https://www.icann.org/stewardship-implementation under "Meetings & Work Sessions" section.
*** Meeting Notes ***
Names SLEs Data Review 21 Jul 16 @ 21:00 UTC
Attendees: Ali Mohammadi
Allan MacGillivray
Chuck Gomes (RySG)
Elaine Pruis - Donuts
Jay Daley
Jeff Neuman
Jeffrey Eckhaus
Kim Davies
Mary Uduma
Naela Sarras
Nathalie Vergnolle
Patricio Poblete
Paul Kane
Trang Nguyen
W Murray
Yuko Green
Notes:
Data report walk-thru: Link to the Names SLEs preliminary data report:
http://mm.icann.org/pipermail/cwg-stewardship/attachments/20160715/92175b9a/...
[Post-meeting note] Description of Change Requests Categories:
* Category I: Routine updates impacting Root Zone File
* Category II: Routine updates not impacting Root Zone File
* Category III: Creating or Transferring a gTLD
* Category IV: Creating or Transferring a ccTLD
* Category V: Other Change Requests
General notes: The purpose of the call was to initiate discussions with the DT-A. It was not intended that thresholds be formally set during this meeting. The recommendations presented in the report were developed by staff, and the data reflects the new SLE data that has been collected since early March 2016, when the corresponding changes in RZMS were deployed. All data collected is in calendar days. It was suggested to add to the report any DT-A recommendations that may have been made in previous discussions. The dashboard will make visible thresholds and actual performance, and will also highlight any missed thresholds. The following notes are specific to each measurement.
Processing Performance (Submission): Support from group for proposed recommendation, which is aligned with DT-A's initial recommendation of 1min (95%).
Time for lodgment of email requests: Group requested that a target be set for this measurement, even in the absence of data. Group suggested threshold of 2 days. Further discussions needed for this threshold.
Time to return results for technical checks following submission of request via automated submission interface: Suggestion to align Category III and IV threshold to 50 min (95%) for both categories.
Time to return results for subsequent performance of technical checks during retesting due to earlier failed tests: No comments from the group.
Time to return results for performance of technical checks during Supplemental Technical Check phase: Suggestion to align Cat. III and IV to 5 min (95%).
Time for authorization contacts to be asked to approve CR: No comments from the group.
Time for response to be affirmed by IANA: No comments from the group.
Time to complete all other validations and reviews by IANA and release request for implementation: Suggested thresholds: Cat I - 5d (90%) Cat II - 5d (90%) Cat III - 5d (90%) Cat IV - Group agreed that the percentage limit should be low. In the absence of data available, threshold could be set to what would be considered a reasonable time to perform the work. This measurement requires further discussion.
There was no time to go through the rest of the measurements. It was agreed that further discussions will be carried out via the mail list and another call scheduled if necessary.
Dashboard demo: A real-time dashboard has been developed and the beta was demonstrated to the group. It will be ready to be deployed by the time of the transition. In addition to presenting the data as customizable charts, the tool also provides the ability to download anonymized raw data to allow for further analysis. It was suggested to add the region or the language of the requester to the log file. ICANN acknowledge that this is a good suggestion, but will need to be implemented as an enhancement post transition. A request was made to share a beta version of the dashboard with the group. This request will be looked into by staff, as authorization from NTIA would be necessary.
Chat History: Yuko Green: Hello, welcome to Names SLEs Data Review meeting!
Paul Kane: Kim - can you remind me your job title
Kim Davies: Director, Technical Services
Paul Kane: Thanks
Jay Daley: Just to clairfy - this document was shared a week ago (15th) not a couple of weeks ago
Chuck Gomes (RySG): I am 1754
Yuko Green: Thank you Chuck!
Yuko Green: Who may be the phone number 1108?
Jeff Neuman: correct
Mary Uduma: no audio
Jeff Neuman: DTA asked for the data so that we could determine the specific metrics
Jeff Neuman: correct
Jeff Neuman: we were hoping to have 6 months of data, but 3 will have to do
Mary Uduma: Audio breaking
Yuko Green: @Mary, we hear everything without breaking up. May I suggest that you try calling back or re-logging back into the room?
Jeffrey Eckhaus: the sound via computer / adobe connect is clear for me
Yuko Green: @Jefferey, thank you for confirming
Jay Daley: Patricio - can you mute your line?
Yuko Green: All, please mute your phone/microphone when not speaking
Elaine Pruis - Donuts: chuck and jeff mute?
Jay Daley: Mary - can you mute your microphone please
Jeffrey Eckhaus: that is why I was so quick to agree to the 60s
Jeffrey Eckhaus: we had already liked that number as a group
Patricio Poblete: I was referring to CWG- DT-A - SLE - for discussion in Istanbul - DRAFT.pdf
Jeff Neuman: you can use business days
Jeff Neuman: 2 business days
Jeff Neuman: I agree with Jay, but not sure this is the SLA where is would matter
Jeff Neuman: Category III seem awfully high
Jeffrey Eckhaus: Hi All , need to drop off a little early, but will catch up on the last 20 minutes or so later today
Elaine Pruis - Donuts: would the host please add the definitions of the categories to teh "notes" section
Elaine Pruis - Donuts: or add a window with the info
Jeff Neuman: I also thing there are ways to account for months were there are fewer transactions. Understand that assigning a percentage is difficult, but that should not preclude setting reasonable targets
Jeff Neuman: Foe example, in a given month if there are 5 applicable transactions, rather than a percentage, you can say that 4 out of the 5 should meet the target
Jay Daley: happy for the demo
Jay Daley: I agree Jeff
Chuck Gomes (RySG): I have to jump off so let me share a few comments: 1) With regard to Root Zone Maintainer thresholds, it is essential that performance definitions are the same as in the RZMA agreement; 2) performance measures should exclude planned outages; 3) as a CWG member and not a member of DT-A, I would really appreciate it if DT-A would eventually present its recommended thresholds to the CWG.
Paul Kane: Compare DTA's Presnted report for SLE with IANA's proposal; No blanks, use historical data, no discrimination between gTLDs and ccTLDs, Need to enter into dialogue with DTA to refine, Seems to be double dipping, Add notes are to categories to help reader
Paul Kane: Exclude NTIA's process time
Jeff Neuman: Good comment Jay
Jeff Neuman: Thanks Kim. I do think it was a good first draft and appreciate the walk through
****************
Thanks, Nathalie.
Direct line: +1-310-578-8957 Mobile: +1-310-938-1037 Skype: nathalie.vergnolle.icann Jabber: nathalie.vergnolle@jabber.icann.org<mailto:nathalie.vergnolle@jabber.i cann.org>
_______________________________________________ dt1 mailing list dt1@icann.org https://mm.icann.org/mailman/listinfo/dt1
Dear members of the DT-A and CWG, IANA staff have now provided proposals for the Names SLEs measurements where no data is available (these had been previously left blank). The attached table has been updated to include those elements (in column E + notes), see attached. Thanks, Nathalie. -----Original Message----- From: Gomes, Chuck [mailto:cgomes@verisign.com] Sent: Thursday, July 28, 2016 6:51 AM To: Nathalie Vergnolle <nathalie.vergnolle@icann.org>; dt1@icann.org; cwg-stewardship@icann.org Subject: RE: [CWG-Stewardship] [SLE Team] FW: Names SLEs Data Review - Meeting Notes (21 July 2016 @ 21:00 UTC) Thanks Nathalie. Chuck -----Original Message----- From: Nathalie Vergnolle [mailto:nathalie.vergnolle@icann.org] Sent: Thursday, July 28, 2016 1:39 AM To: Gomes, Chuck; dt1@icann.org; cwg-stewardship@icann.org Subject: RE: [CWG-Stewardship] [SLE Team] FW: Names SLEs Data Review - Meeting Notes (21 July 2016 @ 21:00 UTC) Hi Chuck, The proposals of the group made on the call have been captured in column F ('DT-A proposal July 16'), plus some notes in the 'Notes' section. So, unless I missed something or incorrectly captured it, the comments made on the call shouldn't need to be repeated. Thanks, Nathalie. -----Original Message----- From: Gomes, Chuck [mailto:cgomes@verisign.com] Sent: Wednesday, July 27, 2016 5:29 PM To: Nathalie Vergnolle <nathalie.vergnolle@icann.org>; dt1@icann.org; cwg-stewardship@icann.org Subject: RE: [CWG-Stewardship] [SLE Team] FW: Names SLEs Data Review - Meeting Notes (21 July 2016 @ 21:00 UTC) Do the comments that were made on the call last week need to be repeated? Chuck -----Original Message----- From: cwg-stewardship-bounces@icann.org [mailto:cwg-stewardship-bounces@icann.org] On Behalf Of Nathalie Vergnolle Sent: Wednesday, July 27, 2016 7:07 PM To: dt1@icann.org; cwg-stewardship@icann.org Subject: Re: [CWG-Stewardship] [SLE Team] FW: Names SLEs Data Review - Meeting Notes (21 July 2016 @ 21:00 UTC) Dear members of the CWG and Names SLEs work group, As agreed during the Names SLEs call held on July 21st, we are seeking additional feedback through the mailing list with regards to the proposal that was circulated on July 15th on Names SLEs. To facilitate the review, I have reformatted the data in a table that presents the various proposals received to-date. Please note that the measurements discussed in Istanbul in March-15 were different from the final proposal of the work group. Some values may have been improperly mapped in the attached table - that would be my own wrong interpretation, and all feedbacks will be appreciated! Kindly asking for all to provide their comments preferably by next Tuesday, August 2nd. A follow-up call will be set shortly thereafter as necessary. Thank you, Nathalie. Link to the July15 proposal: http://mm.icann.org/pipermail/cwg-stewardship/attachments/20160715/92175b9a/... Link to the Istanbul March 2015 document: https://community.icann.org/download/attachments/52891144/CWG-%20DT-A%20-%20... -----Original Message----- From: dt1-bounces@icann.org [mailto:dt1-bounces@icann.org] On Behalf Of Paul M Kane - CWG Sent: Saturday, July 23, 2016 3:04 PM To: Yuko Green <yuko.green@icann.org> Cc: dt1@icann.org; cwg-stewardship@icann.org Subject: Re: [SLE Team] [CWG-Stewardship] FW: Names SLEs Data Review - Meeting Notes (21 July 2016 @ 21:00 UTC) Quick update following initial SLE Data Review. The call (open to all 177 CWG participants) was held on the 21st July, to give an initial overview of the SLE Data captured over April, May and June. The call had 15 attendees and I am pleased to report all DTA members were present. It was a constructive call which lasted 1 hour. Kim Davies, Director, Technical Services, IANA gave an introduction to the collected data captured over the 3 month period. There was a request that IANA staff produce a comparison table between the thresholds initially recommended by DTA and presented to the CWG in Istanbul March 2015 (https://community.icann.org/download/attachments/52891144/CWG-%20DT-A%20-%20...) and the IANA proposed thresholds gathered in the 3 month window. IANA agreed to do this, and there will be a follow up call/email. There was a request that the SLE thresholds proposed by IANA be transposed to the CWG approved format and tabulated as captured in the document IANA Service Level Expectations - APPROVED.pdf (https://community.icann.org/download/attachments/52891144/IANA%20Service%20L...) It was felt that it was inappropriate to have unspecified SLE performance thresholds and so it was suggested that historical data be used to populate data sets where insufficient data was available from the 3 month window. IANA proposed to have different performance expectations between gTLDs and ccTLDs and DTA members prefered that the IANA service standards should have the same thresholds for both ccTLDs and gTLDs. In some scenarios the IANA proposed thresholds seem to include "double dipping" (counting scenarios twice)... and all parties agreed to take a closer look to ensure transparency on performance monitoring. During the data capture window, (which still involved NTIA's review, which post transition will be removed yet) no allowance was made to remove the time taken for NTIA's involvement in the process in the IANA proposed SLE thresholds. As a result as this SLE will refer to the situation without NTIA's role, the request was made remove the NTIA's process time so emulate the SLE as close to the post-transition situation as possible. There was a request to include Advisory Notes to assist all readers disseminate the different Categories of transactions. The SLE specification is far from complete and DTA members agreed to work with IANA in the coming weeks to refine and populate the thresholds. We will be reporting back to the full CWG on a regular basis. We were also given a "preview" of the Dashboard requested by DTA, which conveys performance information pictorially with raw data also being available for custom monitoring .... I am confident the dashboard will be well received when it is formally launched. Best Paul Quoting Yuko Green <yuko.green@icann.org>:
Dear members of the CWG,
Please see below the meeting notes from Names SLEs Data Review meeting that took place on 21 July at 21:00 UTC.
Regards, Yuko Green
From: Nathalie Vergnolle Sent: Friday, July 22, 2016 2:09 PM To: dt1@icann.org Cc: Trang Nguyen <trang.nguyen@icann.org>; Yuko Green <yuko.green@icann.org>; Alireza Mohammadi <alireza.mohammadi@icann.org>; Naela Sarras <naela.sarras@icann.org> Subject: Names SLEs Data Review - Meeting Notes (21 July 2016 @ 21:00 UTC)
Dear members of the Names SLEs work group,
Please see below the meeting notes and chat history from yesterday's Names SLEs Data Review call.
The presentation material, audio and AC room recordings are now posted at https://www.icann.org/stewardship-implementation under "Meetings & Work Sessions" section.
*** Meeting Notes ***
Names SLEs Data Review 21 Jul 16 @ 21:00 UTC
Attendees: Ali Mohammadi
Allan MacGillivray
Chuck Gomes (RySG)
Elaine Pruis - Donuts
Jay Daley
Jeff Neuman
Jeffrey Eckhaus
Kim Davies
Mary Uduma
Naela Sarras
Nathalie Vergnolle
Patricio Poblete
Paul Kane
Trang Nguyen
W Murray
Yuko Green
Notes:
Data report walk-thru: Link to the Names SLEs preliminary data report:
http://mm.icann.org/pipermail/cwg-stewardship/attachments/20160715/92175b9a/...
[Post-meeting note] Description of Change Requests Categories:
* Category I: Routine updates impacting Root Zone File
* Category II: Routine updates not impacting Root Zone File
* Category III: Creating or Transferring a gTLD
* Category IV: Creating or Transferring a ccTLD
* Category V: Other Change Requests
General notes: The purpose of the call was to initiate discussions with the DT-A. It was not intended that thresholds be formally set during this meeting. The recommendations presented in the report were developed by staff, and the data reflects the new SLE data that has been collected since early March 2016, when the corresponding changes in RZMS were deployed. All data collected is in calendar days. It was suggested to add to the report any DT-A recommendations that may have been made in previous discussions. The dashboard will make visible thresholds and actual performance, and will also highlight any missed thresholds. The following notes are specific to each measurement.
Processing Performance (Submission): Support from group for proposed recommendation, which is aligned with DT-A's initial recommendation of 1min (95%).
Time for lodgment of email requests: Group requested that a target be set for this measurement, even in the absence of data. Group suggested threshold of 2 days. Further discussions needed for this threshold.
Time to return results for technical checks following submission of request via automated submission interface: Suggestion to align Category III and IV threshold to 50 min (95%) for both categories.
Time to return results for subsequent performance of technical checks during retesting due to earlier failed tests: No comments from the group.
Time to return results for performance of technical checks during Supplemental Technical Check phase: Suggestion to align Cat. III and IV to 5 min (95%).
Time for authorization contacts to be asked to approve CR: No comments from the group.
Time for response to be affirmed by IANA: No comments from the group.
Time to complete all other validations and reviews by IANA and release request for implementation: Suggested thresholds: Cat I - 5d (90%) Cat II - 5d (90%) Cat III - 5d (90%) Cat IV - Group agreed that the percentage limit should be low. In the absence of data available, threshold could be set to what would be considered a reasonable time to perform the work. This measurement requires further discussion.
There was no time to go through the rest of the measurements. It was agreed that further discussions will be carried out via the mail list and another call scheduled if necessary.
Dashboard demo: A real-time dashboard has been developed and the beta was demonstrated to the group. It will be ready to be deployed by the time of the transition. In addition to presenting the data as customizable charts, the tool also provides the ability to download anonymized raw data to allow for further analysis. It was suggested to add the region or the language of the requester to the log file. ICANN acknowledge that this is a good suggestion, but will need to be implemented as an enhancement post transition. A request was made to share a beta version of the dashboard with the group. This request will be looked into by staff, as authorization from NTIA would be necessary.
Chat History: Yuko Green: Hello, welcome to Names SLEs Data Review meeting!
Paul Kane: Kim - can you remind me your job title
Kim Davies: Director, Technical Services
Paul Kane: Thanks
Jay Daley: Just to clairfy - this document was shared a week ago (15th) not a couple of weeks ago
Chuck Gomes (RySG): I am 1754
Yuko Green: Thank you Chuck!
Yuko Green: Who may be the phone number 1108?
Jeff Neuman: correct
Mary Uduma: no audio
Jeff Neuman: DTA asked for the data so that we could determine the specific metrics
Jeff Neuman: correct
Jeff Neuman: we were hoping to have 6 months of data, but 3 will have to do
Mary Uduma: Audio breaking
Yuko Green: @Mary, we hear everything without breaking up. May I suggest that you try calling back or re-logging back into the room?
Jeffrey Eckhaus: the sound via computer / adobe connect is clear for me
Yuko Green: @Jefferey, thank you for confirming
Jay Daley: Patricio - can you mute your line?
Yuko Green: All, please mute your phone/microphone when not speaking
Elaine Pruis - Donuts: chuck and jeff mute?
Jay Daley: Mary - can you mute your microphone please
Jeffrey Eckhaus: that is why I was so quick to agree to the 60s
Jeffrey Eckhaus: we had already liked that number as a group
Patricio Poblete: I was referring to CWG- DT-A - SLE - for discussion in Istanbul - DRAFT.pdf
Jeff Neuman: you can use business days
Jeff Neuman: 2 business days
Jeff Neuman: I agree with Jay, but not sure this is the SLA where is would matter
Jeff Neuman: Category III seem awfully high
Jeffrey Eckhaus: Hi All , need to drop off a little early, but will catch up on the last 20 minutes or so later today
Elaine Pruis - Donuts: would the host please add the definitions of the categories to teh "notes" section
Elaine Pruis - Donuts: or add a window with the info
Jeff Neuman: I also thing there are ways to account for months were there are fewer transactions. Understand that assigning a percentage is difficult, but that should not preclude setting reasonable targets
Jeff Neuman: Foe example, in a given month if there are 5 applicable transactions, rather than a percentage, you can say that 4 out of the 5 should meet the target
Jay Daley: happy for the demo
Jay Daley: I agree Jeff
Chuck Gomes (RySG): I have to jump off so let me share a few comments: 1) With regard to Root Zone Maintainer thresholds, it is essential that performance definitions are the same as in the RZMA agreement; 2) performance measures should exclude planned outages; 3) as a CWG member and not a member of DT-A, I would really appreciate it if DT-A would eventually present its recommended thresholds to the CWG.
Paul Kane: Compare DTA's Presnted report for SLE with IANA's proposal; No blanks, use historical data, no discrimination between gTLDs and ccTLDs, Need to enter into dialogue with DTA to refine, Seems to be double dipping, Add notes are to categories to help reader
Paul Kane: Exclude NTIA's process time
Jeff Neuman: Good comment Jay
Jeff Neuman: Thanks Kim. I do think it was a good first draft and appreciate the walk through
****************
Thanks, Nathalie.
Direct line: +1-310-578-8957 Mobile: +1-310-938-1037 Skype: nathalie.vergnolle.icann Jabber: nathalie.vergnolle@jabber.icann.org<mailto:nathalie.vergnolle@jabber.i cann.org>
_______________________________________________ dt1 mailing list dt1@icann.org https://mm.icann.org/mailman/listinfo/dt1
As I suggested in the call last week, I suggest that DT-A makes its recommendations and submit them to the CWG for confirmation. Chuck -----Original Message----- From: Nathalie Vergnolle [mailto:nathalie.vergnolle@icann.org] Sent: Thursday, July 28, 2016 3:55 PM To: Gomes, Chuck; dt1@icann.org; cwg-stewardship@icann.org Subject: RE: [CWG-Stewardship] [SLE Team] FW: Names SLEs Data Review - Meeting Notes (21 July 2016 @ 21:00 UTC) Dear members of the DT-A and CWG, IANA staff have now provided proposals for the Names SLEs measurements where no data is available (these had been previously left blank). The attached table has been updated to include those elements (in column E + notes), see attached. Thanks, Nathalie. -----Original Message----- From: Gomes, Chuck [mailto:cgomes@verisign.com] Sent: Thursday, July 28, 2016 6:51 AM To: Nathalie Vergnolle <nathalie.vergnolle@icann.org>; dt1@icann.org; cwg-stewardship@icann.org Subject: RE: [CWG-Stewardship] [SLE Team] FW: Names SLEs Data Review - Meeting Notes (21 July 2016 @ 21:00 UTC) Thanks Nathalie. Chuck -----Original Message----- From: Nathalie Vergnolle [mailto:nathalie.vergnolle@icann.org] Sent: Thursday, July 28, 2016 1:39 AM To: Gomes, Chuck; dt1@icann.org; cwg-stewardship@icann.org Subject: RE: [CWG-Stewardship] [SLE Team] FW: Names SLEs Data Review - Meeting Notes (21 July 2016 @ 21:00 UTC) Hi Chuck, The proposals of the group made on the call have been captured in column F ('DT-A proposal July 16'), plus some notes in the 'Notes' section. So, unless I missed something or incorrectly captured it, the comments made on the call shouldn't need to be repeated. Thanks, Nathalie. -----Original Message----- From: Gomes, Chuck [mailto:cgomes@verisign.com] Sent: Wednesday, July 27, 2016 5:29 PM To: Nathalie Vergnolle <nathalie.vergnolle@icann.org>; dt1@icann.org; cwg-stewardship@icann.org Subject: RE: [CWG-Stewardship] [SLE Team] FW: Names SLEs Data Review - Meeting Notes (21 July 2016 @ 21:00 UTC) Do the comments that were made on the call last week need to be repeated? Chuck -----Original Message----- From: cwg-stewardship-bounces@icann.org [mailto:cwg-stewardship-bounces@icann.org] On Behalf Of Nathalie Vergnolle Sent: Wednesday, July 27, 2016 7:07 PM To: dt1@icann.org; cwg-stewardship@icann.org Subject: Re: [CWG-Stewardship] [SLE Team] FW: Names SLEs Data Review - Meeting Notes (21 July 2016 @ 21:00 UTC) Dear members of the CWG and Names SLEs work group, As agreed during the Names SLEs call held on July 21st, we are seeking additional feedback through the mailing list with regards to the proposal that was circulated on July 15th on Names SLEs. To facilitate the review, I have reformatted the data in a table that presents the various proposals received to-date. Please note that the measurements discussed in Istanbul in March-15 were different from the final proposal of the work group. Some values may have been improperly mapped in the attached table - that would be my own wrong interpretation, and all feedbacks will be appreciated! Kindly asking for all to provide their comments preferably by next Tuesday, August 2nd. A follow-up call will be set shortly thereafter as necessary. Thank you, Nathalie. Link to the July15 proposal: http://mm.icann.org/pipermail/cwg-stewardship/attachments/20160715/92175b9a/... Link to the Istanbul March 2015 document: https://community.icann.org/download/attachments/52891144/CWG-%20DT-A%20-%20... -----Original Message----- From: dt1-bounces@icann.org [mailto:dt1-bounces@icann.org] On Behalf Of Paul M Kane - CWG Sent: Saturday, July 23, 2016 3:04 PM To: Yuko Green <yuko.green@icann.org> Cc: dt1@icann.org; cwg-stewardship@icann.org Subject: Re: [SLE Team] [CWG-Stewardship] FW: Names SLEs Data Review - Meeting Notes (21 July 2016 @ 21:00 UTC) Quick update following initial SLE Data Review. The call (open to all 177 CWG participants) was held on the 21st July, to give an initial overview of the SLE Data captured over April, May and June. The call had 15 attendees and I am pleased to report all DTA members were present. It was a constructive call which lasted 1 hour. Kim Davies, Director, Technical Services, IANA gave an introduction to the collected data captured over the 3 month period. There was a request that IANA staff produce a comparison table between the thresholds initially recommended by DTA and presented to the CWG in Istanbul March 2015 (https://community.icann.org/download/attachments/52891144/CWG-%20DT-A%20-%20...) and the IANA proposed thresholds gathered in the 3 month window. IANA agreed to do this, and there will be a follow up call/email. There was a request that the SLE thresholds proposed by IANA be transposed to the CWG approved format and tabulated as captured in the document IANA Service Level Expectations - APPROVED.pdf (https://community.icann.org/download/attachments/52891144/IANA%20Service%20L...) It was felt that it was inappropriate to have unspecified SLE performance thresholds and so it was suggested that historical data be used to populate data sets where insufficient data was available from the 3 month window. IANA proposed to have different performance expectations between gTLDs and ccTLDs and DTA members prefered that the IANA service standards should have the same thresholds for both ccTLDs and gTLDs. In some scenarios the IANA proposed thresholds seem to include "double dipping" (counting scenarios twice)... and all parties agreed to take a closer look to ensure transparency on performance monitoring. During the data capture window, (which still involved NTIA's review, which post transition will be removed yet) no allowance was made to remove the time taken for NTIA's involvement in the process in the IANA proposed SLE thresholds. As a result as this SLE will refer to the situation without NTIA's role, the request was made remove the NTIA's process time so emulate the SLE as close to the post-transition situation as possible. There was a request to include Advisory Notes to assist all readers disseminate the different Categories of transactions. The SLE specification is far from complete and DTA members agreed to work with IANA in the coming weeks to refine and populate the thresholds. We will be reporting back to the full CWG on a regular basis. We were also given a "preview" of the Dashboard requested by DTA, which conveys performance information pictorially with raw data also being available for custom monitoring .... I am confident the dashboard will be well received when it is formally launched. Best Paul Quoting Yuko Green <yuko.green@icann.org>:
Dear members of the CWG,
Please see below the meeting notes from Names SLEs Data Review meeting that took place on 21 July at 21:00 UTC.
Regards, Yuko Green
From: Nathalie Vergnolle Sent: Friday, July 22, 2016 2:09 PM To: dt1@icann.org Cc: Trang Nguyen <trang.nguyen@icann.org>; Yuko Green <yuko.green@icann.org>; Alireza Mohammadi <alireza.mohammadi@icann.org>; Naela Sarras <naela.sarras@icann.org> Subject: Names SLEs Data Review - Meeting Notes (21 July 2016 @ 21:00 UTC)
Dear members of the Names SLEs work group,
Please see below the meeting notes and chat history from yesterday's Names SLEs Data Review call.
The presentation material, audio and AC room recordings are now posted at https://www.icann.org/stewardship-implementation under "Meetings & Work Sessions" section.
*** Meeting Notes ***
Names SLEs Data Review 21 Jul 16 @ 21:00 UTC
Attendees: Ali Mohammadi
Allan MacGillivray
Chuck Gomes (RySG)
Elaine Pruis - Donuts
Jay Daley
Jeff Neuman
Jeffrey Eckhaus
Kim Davies
Mary Uduma
Naela Sarras
Nathalie Vergnolle
Patricio Poblete
Paul Kane
Trang Nguyen
W Murray
Yuko Green
Notes:
Data report walk-thru: Link to the Names SLEs preliminary data report:
http://mm.icann.org/pipermail/cwg-stewardship/attachments/20160715/92175b9a/...
[Post-meeting note] Description of Change Requests Categories:
* Category I: Routine updates impacting Root Zone File
* Category II: Routine updates not impacting Root Zone File
* Category III: Creating or Transferring a gTLD
* Category IV: Creating or Transferring a ccTLD
* Category V: Other Change Requests
General notes: The purpose of the call was to initiate discussions with the DT-A. It was not intended that thresholds be formally set during this meeting. The recommendations presented in the report were developed by staff, and the data reflects the new SLE data that has been collected since early March 2016, when the corresponding changes in RZMS were deployed. All data collected is in calendar days. It was suggested to add to the report any DT-A recommendations that may have been made in previous discussions. The dashboard will make visible thresholds and actual performance, and will also highlight any missed thresholds. The following notes are specific to each measurement.
Processing Performance (Submission): Support from group for proposed recommendation, which is aligned with DT-A's initial recommendation of 1min (95%).
Time for lodgment of email requests: Group requested that a target be set for this measurement, even in the absence of data. Group suggested threshold of 2 days. Further discussions needed for this threshold.
Time to return results for technical checks following submission of request via automated submission interface: Suggestion to align Category III and IV threshold to 50 min (95%) for both categories.
Time to return results for subsequent performance of technical checks during retesting due to earlier failed tests: No comments from the group.
Time to return results for performance of technical checks during Supplemental Technical Check phase: Suggestion to align Cat. III and IV to 5 min (95%).
Time for authorization contacts to be asked to approve CR: No comments from the group.
Time for response to be affirmed by IANA: No comments from the group.
Time to complete all other validations and reviews by IANA and release request for implementation: Suggested thresholds: Cat I - 5d (90%) Cat II - 5d (90%) Cat III - 5d (90%) Cat IV - Group agreed that the percentage limit should be low. In the absence of data available, threshold could be set to what would be considered a reasonable time to perform the work. This measurement requires further discussion.
There was no time to go through the rest of the measurements. It was agreed that further discussions will be carried out via the mail list and another call scheduled if necessary.
Dashboard demo: A real-time dashboard has been developed and the beta was demonstrated to the group. It will be ready to be deployed by the time of the transition. In addition to presenting the data as customizable charts, the tool also provides the ability to download anonymized raw data to allow for further analysis. It was suggested to add the region or the language of the requester to the log file. ICANN acknowledge that this is a good suggestion, but will need to be implemented as an enhancement post transition. A request was made to share a beta version of the dashboard with the group. This request will be looked into by staff, as authorization from NTIA would be necessary.
Chat History: Yuko Green: Hello, welcome to Names SLEs Data Review meeting!
Paul Kane: Kim - can you remind me your job title
Kim Davies: Director, Technical Services
Paul Kane: Thanks
Jay Daley: Just to clairfy - this document was shared a week ago (15th) not a couple of weeks ago
Chuck Gomes (RySG): I am 1754
Yuko Green: Thank you Chuck!
Yuko Green: Who may be the phone number 1108?
Jeff Neuman: correct
Mary Uduma: no audio
Jeff Neuman: DTA asked for the data so that we could determine the specific metrics
Jeff Neuman: correct
Jeff Neuman: we were hoping to have 6 months of data, but 3 will have to do
Mary Uduma: Audio breaking
Yuko Green: @Mary, we hear everything without breaking up. May I suggest that you try calling back or re-logging back into the room?
Jeffrey Eckhaus: the sound via computer / adobe connect is clear for me
Yuko Green: @Jefferey, thank you for confirming
Jay Daley: Patricio - can you mute your line?
Yuko Green: All, please mute your phone/microphone when not speaking
Elaine Pruis - Donuts: chuck and jeff mute?
Jay Daley: Mary - can you mute your microphone please
Jeffrey Eckhaus: that is why I was so quick to agree to the 60s
Jeffrey Eckhaus: we had already liked that number as a group
Patricio Poblete: I was referring to CWG- DT-A - SLE - for discussion in Istanbul - DRAFT.pdf
Jeff Neuman: you can use business days
Jeff Neuman: 2 business days
Jeff Neuman: I agree with Jay, but not sure this is the SLA where is would matter
Jeff Neuman: Category III seem awfully high
Jeffrey Eckhaus: Hi All , need to drop off a little early, but will catch up on the last 20 minutes or so later today
Elaine Pruis - Donuts: would the host please add the definitions of the categories to teh "notes" section
Elaine Pruis - Donuts: or add a window with the info
Jeff Neuman: I also thing there are ways to account for months were there are fewer transactions. Understand that assigning a percentage is difficult, but that should not preclude setting reasonable targets
Jeff Neuman: Foe example, in a given month if there are 5 applicable transactions, rather than a percentage, you can say that 4 out of the 5 should meet the target
Jay Daley: happy for the demo
Jay Daley: I agree Jeff
Chuck Gomes (RySG): I have to jump off so let me share a few comments: 1) With regard to Root Zone Maintainer thresholds, it is essential that performance definitions are the same as in the RZMA agreement; 2) performance measures should exclude planned outages; 3) as a CWG member and not a member of DT-A, I would really appreciate it if DT-A would eventually present its recommended thresholds to the CWG.
Paul Kane: Compare DTA's Presnted report for SLE with IANA's proposal; No blanks, use historical data, no discrimination between gTLDs and ccTLDs, Need to enter into dialogue with DTA to refine, Seems to be double dipping, Add notes are to categories to help reader
Paul Kane: Exclude NTIA's process time
Jeff Neuman: Good comment Jay
Jeff Neuman: Thanks Kim. I do think it was a good first draft and appreciate the walk through
****************
Thanks, Nathalie.
Direct line: +1-310-578-8957 Mobile: +1-310-938-1037 Skype: nathalie.vergnolle.icann Jabber: nathalie.vergnolle@jabber.icann.org<mailto:nathalie.vergnolle@jabber.i cann.org>
_______________________________________________ dt1 mailing list dt1@icann.org https://mm.icann.org/mailman/listinfo/dt1
participants (4)
-
Gomes, Chuck -
Nathalie Vergnolle -
Paul M Kane - CWG -
Yuko Green