Minor correction. Section 3.4. Second paragraph. instancess should be instances The RCODE distribution is a raw count of the RCODE values observed in responses generated by the RSO’s instancess during the reporting period. Note in particular that this measurement should exclude any DNS response messages that may be sent to a RSI. On Thu, Feb 20, 2020 at 8:15 AM Ray Bellis <ray@isc.org> wrote:

On 20/02/2020 13:07, Andrew McConachie wrote:

...

Dear RSSAC Caucus Members,

Thanks everyone who submitted edits to RSSAC002v4. The review period ended on Feb 17th and I have incorporated all edits received. None of them were particularly major. Section 10.4 has been updated to list the major changes from v3 to v4.

If no comments are received by*Sunday February 23* this document will be forwarded to the RSSAC for voting.

< https://docs.google.com/document/d/1na7VpPYfo4VEBfQMHwix6I2Puohvc1hvwRh0aW1c...

I missed any discussion relating to this change:

"Section 3.5 (The Number of Sources Seen) was amended to remove text stating that the “num-sources-*” measurements are optional and will be reviewed after 3 years."

I don't agree with it, and as an Operator I am technically unable to fully comply with it. We can only produce figures from a subset of our nodes.

Ray Bellis Director of DNS Operations, ISC. _______________________________________________ rssac-caucus mailing list rssac-caucus@icann.org https://mm.icann.org/mailman/listinfo/rssac-caucus

_______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.

On 21/02/2020 10:34, Andrew McConachie wrote:

Hi Ray,

Thanks for raising this concern.

The following text is in this section in RSSAC002v3: "This set of metrics is marked as optional for a 3-year period following the acceptance and publication of version 1 of this document by RSSAC. As experience grows with fine-grained reporting from many operational root-server instances these values can be phased in over this 3-year period. In case experience shows that these values provide little value overall, or constitute a memory exhaustion attack upon monitoring infrastructure, an amendment should be issued by RSSAC to deprecate the documented collection of this data.”

The above paragraph and the metric num-sources-ipv6 have been removed from section 3.5 in the draft of RSSAC002v4.

If the Caucus would like the collection of num-sources-ipv4 and num-sources-ipv6-aggregate to become optional I recommend a single sentence at the end of the section. Something like, “The collection of these two values is optional.”

I'm fine with the removal of the individual IPv6 source count. However it's unclear whether these figures actually do provide any value, and they are not that easy to collate. Roughly, the process we have at ISC is that each instance (where possible) runs a custom-written daemon that records the unique IPs seen in each one hour interval, and then submits that to a central collector over HTTPs. A daily process then reads in every one of those files (thousands of them!) to determine the union of those data sets.

Then in section 6.6 we will need some way to distinguish between num-sources-* with zero counts, and num-sources-* that are not being collected. Suggestions are welcome.

The particular issue I have is not that they are not collected, but that we are only able to report a figure from a subset of our nodes. However I would rather not be collecting the data at all if there's no value to it. Ray

Thanks Ryan. I’ve corrected it in the draft document.

On Feb 21, 2020, at 22:20, Stephenson, Ryan M CIV DISA IE (USA) <ryan.m.stephenson2.civ@mail.mil> wrote:

Unsure if you got the e-mail about section 3.4 use of instancess vs instances.

-----Original Message----- From: rssac-caucus <rssac-caucus-bounces@icann.org> On Behalf Of Andrew McConachie Sent: Friday, February 21, 2020 5:34 AM To: Ray Bellis <ray@isc.org> Cc: rssac-caucus@icann.org Subject: [Non-DoD Source] Re: [RSSAC Caucus] 48 HOUR LAST CALL : RSSAC002v4

All active links contained in this email were disabled. Please verify the identity of the sender, and confirm the authenticity of all links contained within the message prior to copying and pasting the address to a Web browser.

----

...

On Feb 20, 2020, at 14:14, Ray Bellis <ray@isc.org> wrote:

On 20/02/2020 13:07, Andrew McConachie wrote:

...

Dear RSSAC Caucus Members,

Thanks everyone who submitted edits to RSSAC002v4. The review period ended on Feb 17th and I have incorporated all edits received. None of them were particularly major. Section 10.4 has been updated to list the major changes from v3 to v4.

If no comments are received by*Sunday February 23* this document will be forwarded to the RSSAC for voting.

<Caution-https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_documen... >

I missed any discussion relating to this change:

"Section 3.5 (The Number of Sources Seen) was amended to remove text stating that the “num-sources-*” measurements are optional and will be reviewed after 3 years."

I don't agree with it, and as an Operator I am technically unable to fully comply with it. We can only produce figures from a subset of our nodes.

Hi Ray,

Thanks for raising this concern.

The following text is in this section in RSSAC002v3: "This set of metrics is marked as optional for a 3-year period following the acceptance and publication of version 1 of this document by RSSAC. As experience grows with fine-grained reporting from many operational root-server instances these values can be phased in over this 3-year period. In case experience shows that these values provide little value overall, or constitute a memory exhaustion attack upon monitoring infrastructure, an amendment should be issued by RSSAC to deprecate the documented collection of this data.”

The above paragraph and the metric num-sources-ipv6 have been removed from section 3.5 in the draft of RSSAC002v4.

If the Caucus would like the collection of num-sources-ipv4 and num-sources-ipv6-aggregate to become optional I recommend a single sentence at the end of the section. Something like, “The collection of these two values is optional.”

Then in section 6.6 we will need some way to distinguish between num-sources-* with zero counts, and num-sources-* that are not being collected. Suggestions are welcome.

Thanks, Andrew

_______________________________________________ rssac-caucus mailing list rssac-caucus@icann.org Caution-https://mm.icann.org/mailman/listinfo/rssac-caucus

_______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (Caution-https://www.icann.org/privacy/policy) and the website Terms of Service (Caution-https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.

On 21/02/2020 22:33, Paul Hoffman wrote:

Just to be clear: are you saying that you can only produce figures for Section 3.5 (number of sources seen) for a subset of your nodes, or that you can only produce figures for a subset of your nodes for *all* the measurements? If the latter, that's going to mess up the evaluation of the measurements for the whole RSS. Even if it is just the former, that will make the measurements in Section 3.5 be not comparable to the other sections.

Only for the unique sources measurement. Ray

On Mon, 24 Feb 2020 10:45:37 +0000, Andrew McConachie wrote:

...

On Feb 21, 2020, at 23:52, Ray Bellis <ray@isc.org> wrote:

On 21/02/2020 22:33, Paul Hoffman wrote:

...

Just to be clear: are you saying that you can only produce figures for Section 3.5 (number of sources seen) for a subset of your nodes, or that you can only produce figures for a subset of your nodes for *all* the measurements? If the latter, that's going to mess up the evaluation of the measurements for the whole RSS. Even if it is just the former, that will make the measurements in Section 3.5 be not comparable to the other sections.

Only for the unique sources measurement.

Hi Ray,

I changed section 6.6 to mark the ‘unique-sources’ metric as optional. This is reflected in the changelog as well.

This change means that if the 'unique-sources' metric is produced by an RSO they must provide values for both 'num-sources-ipv4' and 'num-sources-ipv6-aggregate’. So produce values for both or produce values for neither.

I would like to hear from other RSO’s if this approach is OK or if a different approach is needed.

Ray asked the important question: is this metric useful. Wearing my "researcher" hat: yes, we have found number of unique-sources in RSSAC data useful to indicate when a RSO is subject to an attack with spoofed sources. But I can understand the challenge in computing this metric. It is by far the most stressful (least compressible). Rather than just declaring the metric optional, what about declaring the precision of the answer variable? Perhaps adding a "fraction-of-traffic-for-sources" field, which for some RSOs would be 1 and others might be less than 1? There is value in a num-sources even if it's based on a 10% sample of traffic, or 10% of anycast sites, and a sampled value might be easier for some RSOs to compute. (And yes, I recognize that sampling could be non-uniform.) -John

Thanks, Andrew

_______________________________________________ rssac-caucus mailing list rssac-caucus@icann.org https://mm.icann.org/mailman/listinfo/rssac-caucus

_______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.

On Tue, 25 Feb 2020 09:32:43 +0000, Andrew McConachie wrote:

...

On Feb 24, 2020, at 21:10, John Heidemann <johnh@isi.edu> wrote:

On Mon, 24 Feb 2020 10:45:37 +0000, Andrew McConachie wrote:

...

...

On Feb 21, 2020, at 23:52, Ray Bellis <ray@isc.org> wrote:

On 21/02/2020 22:33, Paul Hoffman wrote:

...

Just to be clear: are you saying that you can only produce figures for Section 3.5 (number of sources seen) for a subset of your nodes, or that you can only produce figures for a subset of your nodes for *all* the measurements? If the latter, that's going to mess up the evaluation of the measurements for the whole RSS. Even if it is just the former, that will make the measurements in Section 3.5 be not comparable to the other sections.

Only for the unique sources measurement.

Hi Ray,

I changed section 6.6 to mark the ‘unique-sources’ metric as optional. This is reflected in the changelog as well.

This change means that if the 'unique-sources' metric is produced by an RSO they must provide values for both 'num-sources-ipv4' and 'num-sources-ipv6-aggregate’. So produce values for both or produce values for neither.

I would like to hear from other RSO’s if this approach is OK or if a different approach is needed.

Ray asked the important question: is this metric useful.

Wearing my "researcher" hat: yes, we have found number of unique-sources in RSSAC data useful to indicate when a RSO is subject to an attack with spoofed sources.

But I can understand the challenge in computing this metric. It is by far the most stressful (least compressible).

Rather than just declaring the metric optional, what about declaring the precision of the answer variable? Perhaps adding a "fraction-of-traffic-for-sources" field, which for some RSOs would be 1 and others might be less than 1? There is value in a num-sources even if it's based on a 10% sample of traffic, or 10% of anycast sites, and a sampled value might be easier for some RSOs to compute.

(And yes, I recognize that sampling could be non-uniform.)

-John

Hi John,

I think the idea of a percent-instances-sampled makes a lot of sense, I’m just concerned about adding it this late in the process of RSSAC002v4. Is it OK to postpone this suggestion until RSSAC002v5?

I can imagine a fair amount of discussion surrounding how exactly to measure precision given the different operating environments.

That is understandable. I'm certainly not insisting it go in. (Although making unique-sources optional is also a major change.) -John

I'm still having a hard time figuring out what the 'num-sources-ipv6-aggregate' number would actually *mean*, and am somewhat concerned that it will become an attractive nuisance -- people will make assumptions about what the metric means, and will draw dangerously flawed conclusions from this. Recording individual (/128) level metrics is largely meaningless - a perfectly reasonable implementation could use RFC 8273 (Unique IPv6 Prefix per Host) style deployments, and use the source address as additional entropy, or per-port worker bindings - this would result in one recursive server instance looking like O(num_queries). Recording all of the addresses is infeasible, and a metric of 670 used addresses doesn't tell you anything - if could be one server, it could be 670. Recording /64s also seems largely meaningless - some deployments will have a number of different "servers" in the same /64 - this means that if you see a query from 2001:DB8:17:42:86::/64, it could be 1 instance or 670 instances. Unfortunately, while you and I (might!) agree on that, you *know* that if a metric like: num-sources-ipv6-aggregate: 565433 is published, people will immediately look at this and say "there are five hundred sixty-five thousand four hundred thirty-three IPv6 nameservers" and will start graphing this, making predictions about the growth of the internet, etc. Yes, they will be wrong, yes, they shouldn't do this -- but they will. I don't know of a way to stop this, other than expecting everyone to include a comment that says: # The below metric doesn't mean what you think it does, go see: <link to section in RSSACxxx> (I increasingly think that the OpenMetrics / prometheus exposition format should have been what this used - https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/ex... - there are many parsers, and comments are an expected conventions - but that is a separate rathole...) W On Tue, Feb 25, 2020 at 10:06 AM Andrew McConachie <andrew.mcconachie@icann.org> wrote:

On Feb 25, 2020, at 15:32, John Heidemann <johnh@isi.edu> wrote:

On Tue, 25 Feb 2020 09:32:43 +0000, Andrew McConachie wrote:

On Feb 24, 2020, at 21:10, John Heidemann <johnh@isi.edu> wrote:

On Mon, 24 Feb 2020 10:45:37 +0000, Andrew McConachie wrote:

On Feb 21, 2020, at 23:52, Ray Bellis <ray@isc.org> wrote:

On 21/02/2020 22:33, Paul Hoffman wrote:

Just to be clear: are you saying that you can only produce figures for Section 3.5 (number of sources seen) for a subset of your nodes, or that you can only produce figures for a subset of your nodes for *all* the measurements? If the latter, that's going to mess up the evaluation of the measurements for the whole RSS. Even if it is just the former, that will make the measurements in Section 3.5 be not comparable to the other sections.

Only for the unique sources measurement.

Hi Ray,

I changed section 6.6 to mark the ‘unique-sources’ metric as optional. This is reflected in the changelog as well.

This change means that if the 'unique-sources' metric is produced by an RSO they must provide values for both 'num-sources-ipv4' and 'num-sources-ipv6-aggregate’. So produce values for both or produce values for neither.

I would like to hear from other RSO’s if this approach is OK or if a different approach is needed.

Ray asked the important question: is this metric useful.

Wearing my "researcher" hat: yes, we have found number of unique-sources in RSSAC data useful to indicate when a RSO is subject to an attack with spoofed sources.

But I can understand the challenge in computing this metric. It is by far the most stressful (least compressible).

Rather than just declaring the metric optional, what about declaring the precision of the answer variable? Perhaps adding a "fraction-of-traffic-for-sources" field, which for some RSOs would be 1 and others might be less than 1? There is value in a num-sources even if it's based on a 10% sample of traffic, or 10% of anycast sites, and a sampled value might be easier for some RSOs to compute.

(And yes, I recognize that sampling could be non-uniform.)

-John

Hi John,

I think the idea of a percent-instances-sampled makes a lot of sense, I’m just concerned about adding it this late in the process of RSSAC002v4. Is it OK to postpone this suggestion until RSSAC002v5?

I can imagine a fair amount of discussion surrounding how exactly to measure precision given the different operating environments.

That is understandable. I'm certainly not insisting it go in.

(Although making unique-sources optional is also a major change.)

It is considering the diff between the document circulated at the beginning of this thread and the current draft RSSAC002v4. But it’s not a major change between the draft RSSAC002v4 and RSSAC002v3. num-sources-* have been optional since RSSAC002v1.

I’m still interested in hearing from other Caucus members what they think about the draft RSSAC002v4 and if any other changes are needed. <https://docs.google.com/document/d/1na7VpPYfo4VEBfQMHwix6I2Puohvc1hvwRh0aW1c...>

Please let me know by the end of this week if there are any more items to discuss.

Thanks, Andrew

_______________________________________________ rssac-caucus mailing list rssac-caucus@icann.org https://mm.icann.org/mailman/listinfo/rssac-caucus

_______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.

-- I don't think the execution is relevant when it was obviously a bad idea in the first place. This is like putting rabid weasels in your pants, and later expressing regret at having chosen those particular rabid weasels and that pair of pants. ---maf

On Feb 24, 2020, at 01:42, John Heidemann <johnh@isi.edu> wrote:

On Thu, 20 Feb 2020 13:07:27 +0000, Andrew McConachie wrote:

...

Dear RSSAC Caucus Members,

Thanks everyone who submitted edits to RSSAC002v4. The review period ended on Feb 17th and I have incorporated all edits received. None of them were particularly major. Section 10.4 has been updated to list the major changes from v3 to v4.

If no comments are received by Sunday February 23 this document will be forwarded to the RSSAC for voting.

<https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_documen... >

I went over the document.

We have no problem publishing num-sources-ipv6, although v4 no longer requires it.

It took me a while to parse this sentence, but I think I got it. The ‘v4’ in the sentence above refers to RSSAC002v4, NOT IPv4 :)

A strict reading of v4 actually prohibits us from continuing to provide this information.

Two suggestions:

I suggest that section 7 be ammended to add the paragraph.

"In addition, RSOs who wish to public statistics from prior RSSAC002 versions that are no longer required may continue to do so using the historic metrics."

(Although is the term RSO or RSI? Maybe this is a missed change?)

I added, "RSOs who wish to publish statistics from prior versions of this specification, but that are no longer required, may continue to do so using the historic metric name." RSOs publish metrics about RSIs. So I believe it should be RSOs here. However, it’s not clear to me what this means for the unique-sources metric. Would the reporting RSO include num-sources-ipv6 as part of their rssac002v4 unique-sources metric(option A), or would there be a separate rssac002v3 unique-sources with a single key for num-sources-ipv6(option B)? OPTION A ======== version: rssac002v4 service: a.root-servers.net start-period: 2016-01-01T00:00:00Z metric: unique-sources num-sources-ipv4: 3740666 num-sources-ipv6-aggregate: 114142 num-sources-ipv6: 114142 OPTION B ======== version: rssac002v4 service: a.root-servers.net start-period: 2016-01-01T00:00:00Z metric: unique-sources num-sources-ipv4: 3740666 num-sources-ipv6-aggregate: 114142 version: rssac002v3 service: a.root-servers.net start-period: 2016-01-01T00:00:00Z metric: unique-sources num-sources-ipv6: 114142

The reasoning is: (1) we should already make sure we do not reuse historic metric identifiers, so there is no chance of collison. (2) Since metrics are an tag-value list without specified ordering, users of the data should already handle extra names.

Thanks, -John

On Feb 24, 2020, at 12:59, Ray Bellis <ray@isc.org> wrote:

On 24/02/2020 10:17, Andrew McConachie wrote:

...

However, it’s not clear to me what this means for the unique-sources metric. Would the reporting RSO include num-sources-ipv6 as part of their rssac002v4 unique-sources metric(option A), or would there be a separate rssac002v3 unique-sources with a single key for num-sources-ipv6(option B)?

...

OPTION A ======== version: rssac002v4 service: a.root-servers.net start-period: 2016-01-01T00:00:00Z metric: unique-sources num-sources-ipv4: 3740666 num-sources-ipv6-aggregate: 114142 num-sources-ipv6: 114142

OPTION B ======== version: rssac002v4 service: a.root-servers.net start-period: 2016-01-01T00:00:00Z metric: unique-sources num-sources-ipv4: 3740666 num-sources-ipv6-aggregate: 114142

version: rssac002v3 service: a.root-servers.net start-period: 2016-01-01T00:00:00Z metric: unique-sources num-sources-ipv6: 114142

The latter isn't legal YAML if stored in a single file.

Thanks for that.

I'd just go with option A, and ensure that we in future revisions we never change the semantics of any previously specified keys.

Makes sense, and I don’t think this requires any new text in the document to make clear.

Ray

_______________________________________________ rssac-caucus mailing list rssac-caucus@icann.org https://mm.icann.org/mailman/listinfo/rssac-caucus

_______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://urldefense.proofpoint.com/v2/url?u=https-3A__www.icann.org_privacy_p... ) and the website Terms of Service (https://urldefense.proofpoint.com/v2/url?u=https-3A__www.icann.org_privacy_t... ). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.