FOR REVIEW: RSSAC Advisory on Metrics for the DNS Root Servers and Root Server System
Dear RSSAC Caucus, On behalf of the RSSAC Caucus Metrics work party, please find the draft RSSAC Advisory on Metrics for the DNS Root Servers and Root Server System. The work party was formed in January 2019. It is led by Duane Wessels and Russ Mundy. After 12 months of deliberations through teleconferences, workshops, face to face meetings at ICANN and IETF, the draft final advisory is ready. Kindly review the document by 10 January 2020: * for any proposed changes, please send to the caucus list identifying the page number, the original paragraph or sentence, and your proposed edit. * We are also copy-editing the document. For any copy-edit changes, please email it to steve.sheng@icann.org<mailto:steve.sheng@icann.org>. WP members, please pay attention to the first bullet in the introduction, as it has been revised today as a result of the teleconference. Best Steve
On 17/12/2019 18:28, Steve Sheng wrote:
* for any proposed changes, please send to the caucus list identifying the page number, the original paragraph or sentence, and your proposed edit.
Re: the issue with the intro text discussed on the call just now, I propose: "Defines measurements and metrics that could be used to set future Service Level Expectations of the Root Server Operators (RSOs)". Ray
On Dec 17, 2019, at 10:33 AM, Ray Bellis <ray@isc.org> wrote:
On 17/12/2019 18:28, Steve Sheng wrote:
* for any proposed changes, please send to the caucus list identifying the page number, the original paragraph or sentence, and your proposed edit.
Re: the issue with the intro text discussed on the call just now, I propose:
"Defines measurements and metrics that could be used to set future Service Level Expectations of the Root Server Operators (RSOs)".
Hi Ray, And for the benefit of others, the wording under discussion appears in the very first bullet of the Introduction section. The old text was: Defines measurements and metrics to ensure Root Server Operators (RSOs) are meeting a minimum level of performance. It was pointed out (by Ray I believe) that the word "ensure" was maybe problematic given the paragraph later in the section about how none of these are *binding* absent contractual obligations. There was some suggestions made during the call and in the chat. Russ' proposed text included the word "appropriate" which I took some issue with. After the work party call, Russ, Steve, and myself met and we felt that the text below would be best: Defines measurements and metrics that Root Server Operators (RSOs) meet to provide a minimum level of performance. Ray (and everyone) are you okay with that? DW
On Dec 18, 2019, at 7:30 AM, Wessels, Duane via rssac-caucus <rssac-caucus@icann.org> wrote:
And for the benefit of others, the wording under discussion appears in the very first bullet of the Introduction section. The old text was:
Defines measurements and metrics to ensure Root Server Operators (RSOs) are meeting a minimum level of performance.
It was pointed out (by Ray I believe) that the word "ensure" was maybe problematic given the paragraph later in the section about how none of these are *binding* absent contractual obligations. There was some suggestions made during the call and in the chat. Russ' proposed text included the word "appropriate" which I took some issue with. After the work party call, Russ, Steve, and myself met and we felt that the text below would be best:
Defines measurements and metrics that Root Server Operators (RSOs) meet to provide a minimum level of performance.
Ray (and everyone) are you okay with that?
Wearing a "good use of terminology" hat: - RSOs do not meet measurements, and they don't meet metrics: they meet thresholds. - As the rest of the paragraph says, the thresholds are also for non-performance things like correctness. There are similar problems in the rest of the paragraph as well, such as not talking about the RSS, not defining what "RSS" stands for, and mixing up measurements and metrics. A proposed change for the whole paragraph: Defines thresholds that Root Server Operators (RSOs) and the Root Server System (RSS) meet to provide a minimum level of acceptable service. These thresholds are based on metrics designed to assess the performance, availability, and quality of service that each RSO provides. These thresholds, as well as the metrics on which they are based, are included as the RSSAC’s input to a yet-to-be defined evaluation process for future RSOs. --Paul Hoffman
On Dec 19, 2019, at 12:03 PM, Paul Hoffman <paul.hoffman@icann.org> wrote:
On Dec 18, 2019, at 7:30 AM, Wessels, Duane via rssac-caucus <rssac-caucus@icann.org> wrote:
And for the benefit of others, the wording under discussion appears in the very first bullet of the Introduction section. The old text was:
Defines measurements and metrics to ensure Root Server Operators (RSOs) are meeting a minimum level of performance.
It was pointed out (by Ray I believe) that the word "ensure" was maybe problematic given the paragraph later in the section about how none of these are *binding* absent contractual obligations. There was some suggestions made during the call and in the chat. Russ' proposed text included the word "appropriate" which I took some issue with. After the work party call, Russ, Steve, and myself met and we felt that the text below would be best:
Defines measurements and metrics that Root Server Operators (RSOs) meet to provide a minimum level of performance.
Ray (and everyone) are you okay with that?
Wearing a "good use of terminology" hat:
- RSOs do not meet measurements, and they don't meet metrics: they meet thresholds.
- As the rest of the paragraph says, the thresholds are also for non-performance things like correctness.
There are similar problems in the rest of the paragraph as well, such as not talking about the RSS, not defining what "RSS" stands for, and mixing up measurements and metrics. A proposed change for the whole paragraph:
Defines thresholds that Root Server Operators (RSOs) and the Root Server System (RSS) meet to provide a minimum level of acceptable service. These thresholds are based on metrics designed to assess the performance, availability, and quality of service that each RSO provides. These thresholds, as well as the metrics on which they are based, are included as the RSSAC’s input to a yet-to-be defined evaluation process for future RSOs.
Thanks Paul, How does this sound to you? • Defines measurements, metrics, and thresholds that Root Server Operators (RSOs) meet to provide a minimum level of performance. The thresholds are based on technical metrics designed to assess the performance, availability, and quality of service that each RSO provides. The thresholds and the metrics on which they are based are included as the RSSAC’s input to a yet-to-be defined evaluation process for future RSOs. • Defines system-wide, externally verifiable metrics and thresholds which demonstrate that the RSS as a whole is online and serving correct and timely responses. This is mostly your proposed text except that I would like to have "measurements, metrics, and thresholds" in the opening sentence because that represents the flow of how these were designed. i.e. measurements -> metrics -> thresholds. I added "and thresholds" to the second bullet. You can also see these as suggested changes in the google doc. DW
On Jan 3, 2020, at 1:07 PM, Wessels, Duane <dwessels@verisign.com> wrote:
How does this sound to you?
• Defines measurements, metrics, and thresholds that Root Server Operators (RSOs) meet to provide a minimum level of performance. The thresholds are based on technical metrics designed to assess the performance, availability, and quality of service that each RSO provides. The thresholds and the metrics on which they are based are included as the RSSAC’s input to a yet-to-be defined evaluation process for future RSOs.
• Defines system-wide, externally verifiable metrics and thresholds which demonstrate that the RSS as a whole is online and serving correct and timely responses.
Thanks! This is fine except you have to spell out RSS in the second bullet because it's the first use in the document. --Paul Hoffman
participants (4)
-
Paul Hoffman -
Ray Bellis -
Steve Sheng -
Wessels, Duane