Some very different early results looking at publication latency
Hi there. If you read my last message, you saw some data at the bottom. It turns out the data was all wrong. Duane caught the fact that some of it was clearly wrong, and after a bunch of futzing with my report program, I saw the error (hint: time gaps are hard). The new data is below.
From the output, you can see that it is not always the same RSO that is lagging. None of the SOA transitions of the four so far took place in a single five-minute window.
Looking at the individual measurements, some collectors see different SOAs from the same RSO in the same collection period between v4 and v6; this could be routing differences, Some collectors see different SOAS from the same RSO in the same collection period between TCP and UDP; this is possibly due to load balancers and some of the systems behind them not being updated at the same time. I hope this helps. --Paul Hoffman There are 213354 result records in the database Out of 4638 five-minute gaps, 12 have more than one SOA ========== 20190902:221500 (1567487700): a 2019090201: 30 2019090300: 02 b 2019090201: 32 2019090300: 00 c 2019090201: 32 2019090300: 00 d 2019090201: 32 2019090300: 00 e 2019090201: 32 2019090300: 00 f 2019090201: 32 2019090300: 00 g 2019090201: 32 2019090300: 00 h 2019090201: 28 2019090300: 04 i 2019090201: 32 2019090300: 00 j 2019090201: 28 2019090300: 04 k 2019090201: 30 2019090300: 02 l 2019090201: 32 2019090300: 00 m 2019090201: 30 2019090300: 02 20190902:222000 (1567488000): a 2019090201: 00 2019090300: 32 b 2019090201: 00 2019090300: 32 c 2019090201: 00 2019090300: 32 d 2019090201: 00 2019090300: 32 e 2019090201: 00 2019090300: 32 f 2019090201: 04 2019090300: 28 g 2019090201: 00 2019090300: 30 h 2019090201: 00 2019090300: 32 i 2019090201: 00 2019090300: 32 j 2019090201: 00 2019090300: 32 k 2019090201: 01 2019090300: 31 l 2019090201: 25 2019090300: 07 m 2019090201: 00 2019090300: 32 ========== 20190903:101500 (1567530900): a 2019090300: 00 2019090301: 32 b 2019090300: 00 2019090301: 32 c 2019090300: 00 2019090301: 30 d 2019090300: 04 2019090301: 28 e 2019090300: 00 2019090301: 32 f 2019090300: 04 2019090301: 28 g 2019090300: 00 2019090301: 31 h 2019090300: 00 2019090301: 32 i 2019090300: 03 2019090301: 28 j 2019090300: 00 2019090301: 31 k 2019090300: 01 2019090301: 30 l 2019090300: 13 2019090301: 19 m 2019090300: 00 2019090301: 32 20190903:102000 (1567531200): a 2019090300: 00 2019090301: 32 b 2019090300: 00 2019090301: 32 c 2019090300: 00 2019090301: 32 d 2019090300: 00 2019090301: 32 e 2019090300: 00 2019090301: 32 f 2019090300: 04 2019090301: 28 g 2019090300: 00 2019090301: 32 h 2019090300: 00 2019090301: 32 i 2019090300: 00 2019090301: 32 j 2019090300: 00 2019090301: 32 k 2019090300: 00 2019090301: 32 l 2019090300: 00 2019090301: 32 m 2019090300: 00 2019090301: 32 ========== 20190903:222000 (1567574400): a 2019090301: 00 2019090400: 32 b 2019090301: 10 2019090400: 22 c 2019090301: 25 2019090400: 06 d 2019090301: 04 2019090400: 28 e 2019090301: 01 2019090400: 31 f 2019090301: 32 2019090400: 00 g 2019090301: 01 2019090400: 29 h 2019090301: 12 2019090400: 20 i 2019090301: 28 2019090400: 02 j 2019090301: 00 2019090400: 32 k 2019090301: 30 2019090400: 02 l 2019090301: 29 2019090400: 02 m 2019090301: 00 2019090400: 32 20190903:222500 (1567574700): a 2019090301: 00 2019090400: 32 b 2019090301: 00 2019090400: 32 c 2019090301: 00 2019090400: 32 d 2019090301: 00 2019090400: 32 e 2019090301: 00 2019090400: 32 f 2019090301: 04 2019090400: 28 g 2019090301: 00 2019090400: 31 h 2019090301: 00 2019090400: 32 i 2019090301: 00 2019090400: 32 j 2019090301: 00 2019090400: 32 k 2019090301: 01 2019090400: 31 l 2019090301: 01 2019090400: 31 m 2019090301: 00 2019090400: 32 20190903:223000 (1567575000): a 2019090301: 00 2019090400: 32 b 2019090301: 00 2019090400: 32 c 2019090301: 00 2019090400: 32 d 2019090301: 00 2019090400: 32 e 2019090301: 00 2019090400: 32 f 2019090301: 04 2019090400: 28 g 2019090301: 00 2019090400: 31 h 2019090301: 00 2019090400: 32 i 2019090301: 00 2019090400: 32 j 2019090301: 00 2019090400: 32 k 2019090301: 00 2019090400: 32 l 2019090301: 00 2019090400: 32 m 2019090301: 00 2019090400: 32 20190903:223500 (1567575300): a 2019090301: 00 2019090400: 32 b 2019090301: 00 2019090400: 32 c 2019090301: 00 2019090400: 32 d 2019090301: 00 2019090400: 32 e 2019090301: 00 2019090400: 32 f 2019090301: 04 2019090400: 28 g 2019090301: 00 2019090400: 29 h 2019090301: 00 2019090400: 32 i 2019090301: 00 2019090400: 32 j 2019090301: 00 2019090400: 31 k 2019090301: 00 2019090400: 32 l 2019090301: 00 2019090400: 32 m 2019090301: 00 2019090400: 32 20190903:224000 (1567575600): a 2019090301: 00 2019090400: 32 b 2019090301: 00 2019090400: 32 c 2019090301: 00 2019090400: 32 d 2019090301: 00 2019090400: 32 e 2019090301: 00 2019090400: 32 f 2019090301: 04 2019090400: 28 g 2019090301: 00 2019090400: 30 h 2019090301: 00 2019090400: 32 i 2019090301: 00 2019090400: 31 j 2019090301: 00 2019090400: 32 k 2019090301: 00 2019090400: 32 l 2019090301: 00 2019090400: 32 m 2019090301: 00 2019090400: 32 ========== 20190904:081500 (1567610100): a 2019090400: 00 2019090401: 32 b 2019090400: 00 2019090401: 32 c 2019090400: 00 2019090401: 32 d 2019090400: 00 2019090401: 32 e 2019090400: 00 2019090401: 32 f 2019090400: 04 2019090401: 28 g 2019090400: 00 2019090401: 30 h 2019090400: 00 2019090401: 32 i 2019090400: 01 2019090401: 27 j 2019090400: 00 2019090401: 32 k 2019090400: 00 2019090401: 31 l 2019090400: 18 2019090401: 14 m 2019090400: 00 2019090401: 32 20190904:082000 (1567610400): a 2019090400: 00 2019090401: 32 b 2019090400: 00 2019090401: 32 c 2019090400: 00 2019090401: 32 d 2019090400: 00 2019090401: 32 e 2019090400: 00 2019090401: 32 f 2019090400: 04 2019090401: 28 g 2019090400: 00 2019090401: 31 h 2019090400: 00 2019090401: 32 i 2019090400: 01 2019090401: 28 j 2019090400: 00 2019090401: 32 k 2019090400: 00 2019090401: 31 l 2019090400: 00 2019090401: 32 m 2019090400: 00 2019090401: 32 20190904:082500 (1567610700): a 2019090400: 00 2019090401: 32 b 2019090400: 00 2019090401: 32 c 2019090400: 00 2019090401: 32 d 2019090400: 00 2019090401: 32 e 2019090400: 00 2019090401: 32 f 2019090400: 00 2019090401: 32 g 2019090400: 00 2019090401: 29 h 2019090400: 00 2019090401: 32 i 2019090400: 02 2019090401: 27 j 2019090400: 00 2019090401: 32 k 2019090400: 00 2019090401: 32 l 2019090400: 00 2019090401: 31 m 2019090400: 00 2019090401: 32
Hello, Thank you for sharing interesting data. I would like to clarify how to read the data.
20190902:221500 (1567487700): [snip] m 2019090201: 30 2019090300: 02
Does it mean there are 32 vantage points in total, and at unix time 1567487700, 30 of them got SOA serial 2019090201 and the rest two of them got 2019090300, ...
20190902:222002 (1567488002): [snip] m 2019090201: 00 2019090300: 32
... then queried again at unix time 1567488002 which is 302 seconds later from the previous, all of 32 vantage points got SOA serial 2019090300? If I interpret it correctly, focusing on M-Root, it means all server instances receive updated root zone data within 5 minutes. It seems to be consistent with RSSAC 002 load-time metrics [*]. [*] https://rssac.wide.ad.jp/rssac002-metrics/2019/09/load-time/ (please keep in mind the metric is 95%-tile) Best Regards, -- Yoshitaka Aharen <aharen@jprs.co.jp> Japan Registry Services Co., Ltd. On Thu, 5 Sep 2019 02:33:17 +0000 Paul Hoffman <paul.hoffman@icann.org> wrote:
Hi there. If you read my last message, you saw some data at the bottom. It turns out the data was all wrong. Duane caught the fact that some of it was clearly wrong, and after a bunch of futzing with my report program, I saw the error (hint: time gaps are hard). The new data is below.
From the output, you can see that it is not always the same RSO that is lagging. None of the SOA transitions of the four so far took place in a single five-minute window.
Looking at the individual measurements, some collectors see different SOAs from the same RSO in the same collection period between v4 and v6; this could be routing differences, Some collectors see different SOAS from the same RSO in the same collection period between TCP and UDP; this is possibly due to load balancers and some of the systems behind them not being updated at the same time.
I hope this helps.
--Paul Hoffman
There are 213354 result records in the database Out of 4638 five-minute gaps, 12 have more than one SOA
==========
20190902:221500 (1567487700): a 2019090201: 30 2019090300: 02 b 2019090201: 32 2019090300: 00 c 2019090201: 32 2019090300: 00 d 2019090201: 32 2019090300: 00 e 2019090201: 32 2019090300: 00 f 2019090201: 32 2019090300: 00 g 2019090201: 32 2019090300: 00 h 2019090201: 28 2019090300: 04 i 2019090201: 32 2019090300: 00 j 2019090201: 28 2019090300: 04 k 2019090201: 30 2019090300: 02 l 2019090201: 32 2019090300: 00 m 2019090201: 30 2019090300: 02
20190902:222000 (1567488000): a 2019090201: 00 2019090300: 32 b 2019090201: 00 2019090300: 32 c 2019090201: 00 2019090300: 32 d 2019090201: 00 2019090300: 32 e 2019090201: 00 2019090300: 32 f 2019090201: 04 2019090300: 28 g 2019090201: 00 2019090300: 30 h 2019090201: 00 2019090300: 32 i 2019090201: 00 2019090300: 32 j 2019090201: 00 2019090300: 32 k 2019090201: 01 2019090300: 31 l 2019090201: 25 2019090300: 07 m 2019090201: 00 2019090300: 32
==========
20190903:101500 (1567530900): a 2019090300: 00 2019090301: 32 b 2019090300: 00 2019090301: 32 c 2019090300: 00 2019090301: 30 d 2019090300: 04 2019090301: 28 e 2019090300: 00 2019090301: 32 f 2019090300: 04 2019090301: 28 g 2019090300: 00 2019090301: 31 h 2019090300: 00 2019090301: 32 i 2019090300: 03 2019090301: 28 j 2019090300: 00 2019090301: 31 k 2019090300: 01 2019090301: 30 l 2019090300: 13 2019090301: 19 m 2019090300: 00 2019090301: 32
20190903:102000 (1567531200): a 2019090300: 00 2019090301: 32 b 2019090300: 00 2019090301: 32 c 2019090300: 00 2019090301: 32 d 2019090300: 00 2019090301: 32 e 2019090300: 00 2019090301: 32 f 2019090300: 04 2019090301: 28 g 2019090300: 00 2019090301: 32 h 2019090300: 00 2019090301: 32 i 2019090300: 00 2019090301: 32 j 2019090300: 00 2019090301: 32 k 2019090300: 00 2019090301: 32 l 2019090300: 00 2019090301: 32 m 2019090300: 00 2019090301: 32
==========
20190903:222000 (1567574400): a 2019090301: 00 2019090400: 32 b 2019090301: 10 2019090400: 22 c 2019090301: 25 2019090400: 06 d 2019090301: 04 2019090400: 28 e 2019090301: 01 2019090400: 31 f 2019090301: 32 2019090400: 00 g 2019090301: 01 2019090400: 29 h 2019090301: 12 2019090400: 20 i 2019090301: 28 2019090400: 02 j 2019090301: 00 2019090400: 32 k 2019090301: 30 2019090400: 02 l 2019090301: 29 2019090400: 02 m 2019090301: 00 2019090400: 32
20190903:222500 (1567574700): a 2019090301: 00 2019090400: 32 b 2019090301: 00 2019090400: 32 c 2019090301: 00 2019090400: 32 d 2019090301: 00 2019090400: 32 e 2019090301: 00 2019090400: 32 f 2019090301: 04 2019090400: 28 g 2019090301: 00 2019090400: 31 h 2019090301: 00 2019090400: 32 i 2019090301: 00 2019090400: 32 j 2019090301: 00 2019090400: 32 k 2019090301: 01 2019090400: 31 l 2019090301: 01 2019090400: 31 m 2019090301: 00 2019090400: 32
20190903:223000 (1567575000): a 2019090301: 00 2019090400: 32 b 2019090301: 00 2019090400: 32 c 2019090301: 00 2019090400: 32 d 2019090301: 00 2019090400: 32 e 2019090301: 00 2019090400: 32 f 2019090301: 04 2019090400: 28 g 2019090301: 00 2019090400: 31 h 2019090301: 00 2019090400: 32 i 2019090301: 00 2019090400: 32 j 2019090301: 00 2019090400: 32 k 2019090301: 00 2019090400: 32 l 2019090301: 00 2019090400: 32 m 2019090301: 00 2019090400: 32
20190903:223500 (1567575300): a 2019090301: 00 2019090400: 32 b 2019090301: 00 2019090400: 32 c 2019090301: 00 2019090400: 32 d 2019090301: 00 2019090400: 32 e 2019090301: 00 2019090400: 32 f 2019090301: 04 2019090400: 28 g 2019090301: 00 2019090400: 29 h 2019090301: 00 2019090400: 32 i 2019090301: 00 2019090400: 32 j 2019090301: 00 2019090400: 31 k 2019090301: 00 2019090400: 32 l 2019090301: 00 2019090400: 32 m 2019090301: 00 2019090400: 32
20190903:224000 (1567575600): a 2019090301: 00 2019090400: 32 b 2019090301: 00 2019090400: 32 c 2019090301: 00 2019090400: 32 d 2019090301: 00 2019090400: 32 e 2019090301: 00 2019090400: 32 f 2019090301: 04 2019090400: 28 g 2019090301: 00 2019090400: 30 h 2019090301: 00 2019090400: 32 i 2019090301: 00 2019090400: 31 j 2019090301: 00 2019090400: 32 k 2019090301: 00 2019090400: 32 l 2019090301: 00 2019090400: 32 m 2019090301: 00 2019090400: 32
==========
20190904:081500 (1567610100): a 2019090400: 00 2019090401: 32 b 2019090400: 00 2019090401: 32 c 2019090400: 00 2019090401: 32 d 2019090400: 00 2019090401: 32 e 2019090400: 00 2019090401: 32 f 2019090400: 04 2019090401: 28 g 2019090400: 00 2019090401: 30 h 2019090400: 00 2019090401: 32 i 2019090400: 01 2019090401: 27 j 2019090400: 00 2019090401: 32 k 2019090400: 00 2019090401: 31 l 2019090400: 18 2019090401: 14 m 2019090400: 00 2019090401: 32
20190904:082000 (1567610400): a 2019090400: 00 2019090401: 32 b 2019090400: 00 2019090401: 32 c 2019090400: 00 2019090401: 32 d 2019090400: 00 2019090401: 32 e 2019090400: 00 2019090401: 32 f 2019090400: 04 2019090401: 28 g 2019090400: 00 2019090401: 31 h 2019090400: 00 2019090401: 32 i 2019090400: 01 2019090401: 28 j 2019090400: 00 2019090401: 32 k 2019090400: 00 2019090401: 31 l 2019090400: 00 2019090401: 32 m 2019090400: 00 2019090401: 32
20190904:082500 (1567610700): a 2019090400: 00 2019090401: 32 b 2019090400: 00 2019090401: 32 c 2019090400: 00 2019090401: 32 d 2019090400: 00 2019090401: 32 e 2019090400: 00 2019090401: 32 f 2019090400: 00 2019090401: 32 g 2019090400: 00 2019090401: 29 h 2019090400: 00 2019090401: 32 i 2019090400: 02 2019090401: 27 j 2019090400: 00 2019090401: 32 k 2019090400: 00 2019090401: 32 l 2019090400: 00 2019090401: 31 m 2019090400: 00 2019090401: 32
_______________________________________________ rssac-caucus mailing list rssac-caucus@icann.org https://mm.icann.org/mailman/listinfo/rssac-caucus
_______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.
Sorry for the delayed reply. On Sep 9, 2019, at 12:21 AM, Yoshitaka Aharen <aharen@jprs.co.jp> wrote:
I would like to clarify how to read the data.
20190902:221500 (1567487700): [snip] m 2019090201: 30 2019090300: 02
Does it mean there are 32 vantage points in total, and at unix time 1567487700, 30 of them got SOA serial 2019090201 and the rest two of them got 2019090300, ...
Close, but not exact. There are eight vantage points (VPs), and each VP does four tests for availability / response latency (v4udp, v4tcp, v6udp, v6tcp). That's what goes into the 32 total. But you are correct that of those, 30 got the lower serial number and two got the higher.
20190902:222002 (1567488002): [snip] m 2019090201: 00 2019090300: 32
... then queried again at unix time 1567488002 which is 302 seconds later from the previous, all of 32 vantage points got SOA serial 2019090300?
Yes.
If I interpret it correctly, focusing on M-Root, it means all server instances receive updated root zone data within 5 minutes.
Yes. --Paul Hoffman
Hi, all, In order to scale the DNS root service, BGP anycast is widely used and then multiple mirror sites of a root server can be “copied” to different locations and different networks. However, from this aspect, the deployment of DNS root service is not so independent and its scalability will be affected by the BGP. For example, even a mirror site is deployed, the nearby resolution may reach other faraway node due to the decision of BGP (although yes, the geolocation is different from the network topology). Maybe the “local” mirror site can be deployed to attract more resolution traffic in the local network and the performance of overall DNS root service can still be improved (no matter of the deployed locations) from the global perspective, some metrics and measurements are still needed to optimize the deployment locations of the mirror sites. In this way, the performance of DNS root service and the global ability to defend the DDoS to DNS root service can be improved. This is a complex problem because the BGP is dynamic and the networking performance is also dynamic, but I think it is an interesting topic and valuable for the planning of new mirror sites deployments. BR, Zhiwei YAN Zhiwei
On Tuesday, 17 September 2019 03:23:00 UTC YAN Zhiwei wrote:
... For example, even a mirror site is deployed, the nearby resolution may reach other faraway node due to the decision of BGP (although yes, the geolocation is different from the network topology). Maybe the “local” mirror site can be deployed to attract more resolution traffic in the local network and the performance of overall DNS root service can still be improved (no matter of the deployed locations) from the global perspective, some metrics and measurements are still needed to optimize the deployment locations of the mirror sites. ...
zhiwei, the best current practice for anycast operations is RFC 4768. sections 4.3 and 4.4 are especially pointful. does your operational experience or laboratory prototyping/testing reveal any gaps or outdated concepts? my own observations of global cache miss traffic show me that the root zone is not statistically important, and i predict that more good can be done at least total cost of complexity by getting DNSSEC validation and QNAME minimization deployed in more recursive servers ("full resolvers") than by adding more nodes or more letters to the RSS, or by improving their local reachability. -- Paul
Thank you so much, Dear Paul. As you said, RFC4786 is a very good document for the operation and configuration of Anycast service. However, how to decide the location and network of the node (DNS root mirror sites) is somewhat general: In Section 4.2: In general, node placement decisions should be made with consideration of likely traffic requirements, the potential for flash crowds or denial-of-service traffic, the stability of the local routing system, and the failure modes with respect to node failure or local routing system failure. In addition, it seems that the RSO needs to have more communications and understandings with the local operator to schedule the location and network of the deployed mirror site in order to optimize the overall performance of the root service and also fulfill the local requirements. From another aspect about the root service, I totally agree with you that the community can widely use some localization schemes (such as RFC7706, RFC8198...) and these schemes are more scalable than just add the mirror sites or new letters. YAN Zhiwei From: Paul Vixie Date: 2019-09-17 17:50 To: rssac-caucus Subject: Re: [RSSAC Caucus] Geolotation and BGP influence on the mirror site On Tuesday, 17 September 2019 03:23:00 UTC YAN Zhiwei wrote:
... For example, even a mirror site is deployed, the nearby resolution may reach other faraway node due to the decision of BGP (although yes, the geolocation is different from the network topology). Maybe the “local” mirror site can be deployed to attract more resolution traffic in the local network and the performance of overall DNS root service can still be improved (no matter of the deployed locations) from the global perspective, some metrics and measurements are still needed to optimize the deployment locations of the mirror sites. ...
zhiwei, the best current practice for anycast operations is RFC 4768. sections 4.3 and 4.4 are especially pointful. does your operational experience or laboratory prototyping/testing reveal any gaps or outdated concepts? my own observations of global cache miss traffic show me that the root zone is not statistically important, and i predict that more good can be done at least total cost of complexity by getting DNSSEC validation and QNAME minimization deployed in more recursive servers ("full resolvers") than by adding more nodes or more letters to the RSS, or by improving their local reachability. -- Paul _______________________________________________ rssac-caucus mailing list rssac-caucus@icann.org https://mm.icann.org/mailman/listinfo/rssac-caucus _______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.
participants (4)
-
Paul Hoffman -
Paul Vixie -
YAN Zhiwei -
Yoshitaka Aharen