Increased DNSKEY queries to the root servers since the KSK-2010 revocation
Greetings again. Soon after the new DNSKEY RRset was published at 1400 UTC on 11 January 2019 with KSK-2010 revoked, there was a noticeable increase in ./IN/DNSKEY queries sent to root servers. While we have heard of no DNS service interruption for users, the average DNSKEY query load over all servers has more than doubled to between 2% and 2.5%. The increase was quite varied between the different root servers; some experienced almost no increase at all, while others experience an increase of up to 5%. Although some resolvers are newly making rapid queries to the root servers for the root’s DNSKEY RRset, we see no indication that those resolvers are failing to answer DNS queries from their customers. We will continue to monitor the situation. The ICANN organization is evaluating the traffic at the L-root to try to characterize the resolvers that are rapidly asking for the root’s DNSKEY RRset. From this, we can determine if we are able to help the operators of those resolvers to remediate this anomalous behavior. We will also share data with other root server operators who are conducting similar investigations. The results will be reported when our analysis is complete. --Paul Hoffman
Paul, I can share a few details and what we're seeing for A & J root at Verisign. The attached graph shows the daily volume of ./IN/DNKSEY queries we received. There's an increase at the rollover and another at revocation. Pre-rollover we were at about 15M/day and now we're at 275M/day. We identified a few ASNs whose sources send high rates of DNSKEY queries and asked them if they could shed any light. One responded quickly that at least some of their sources were VMs running CentOS 6.7 and BIND 9.8.2. We didn't get any config files but I would bet good money that they're using trusted-keys. DW
Does this have anything to do with the observation that A and J truncate large IPv6 UDP responses and the dual sigs cause a truncated response in IPv6 from root servers A and J which triggers a followup TCP query? (as do B, G and I). (I also note that B and G truncates IPv4 UDP responses at 1280 octets as well (or they did last I looked) Geoff
On 16 Jan 2019, at 7:46 am, Wessels, Duane via ksk-rollover <ksk-rollover@icann.org> wrote:
Paul,
I can share a few details and what we're seeing for A & J root at Verisign. The attached graph shows the daily volume of ./IN/DNKSEY queries we received. There's an increase at the rollover and another at revocation. Pre-rollover we were at about 15M/day and now we're at 275M/day.
We identified a few ASNs whose sources send high rates of DNSKEY queries and asked them if they could shed any light. One responded quickly that at least some of their sources were VMs running CentOS 6.7 and BIND 9.8.2. We didn't get any config files but I would bet good money that they're using trusted-keys.
DW
<rate-of-dot-dnskey-queries.png>_______________________________________________ ksk-rollover mailing list ksk-rollover@icann.org https://mm.icann.org/mailman/listinfo/ksk-rollover
On Jan 15 2019, Paul Hoffman wrote:
Greetings again. Soon after the new DNSKEY RRset was published at 1400 UTC on 11 January 2019 with KSK-2010 revoked, there was a noticeable increase in ./IN/DNSKEY queries sent to root servers.
[much detail omitted here]
The ICANN organization is evaluating the traffic at the L-root to try to characterize the resolvers that are rapidly asking for the root's DNSKEY RRset. From this, we can determine if we are able to help the operators of those resolvers to remediate this anomalous behavior. We will also share data with other root server operators who are conducting similar investigations. The results will be reported when our analysis is complete.
A month on, is there anything more known about this? In particular, for how long did the increased DNSKEY query rate continue? -- Chris Thompson Email: cet1@cam.ac.uk
I gave a presentation on this at FOSDEM. https://fosdem.org/2019/schedule/event/dns_ksk_2010_revoke_monitoring/ There is still no decline. There is a slight increase since the revoke. Roy
On 12 Feb 2019, at 16:52, Chris Thompson <cet1@cam.ac.uk> wrote:
On Jan 15 2019, Paul Hoffman wrote:
Greetings again. Soon after the new DNSKEY RRset was published at 1400 UTC on 11 January 2019 with KSK-2010 revoked, there was a noticeable increase in ./IN/DNSKEY queries sent to root servers.
[much detail omitted here]
The ICANN organization is evaluating the traffic at the L-root to try to characterize the resolvers that are rapidly asking for the root's DNSKEY RRset. From this, we can determine if we are able to help the operators of those resolvers to remediate this anomalous behavior. We will also share data with other root server operators who are conducting similar investigations. The results will be reported when our analysis is complete.
A month on, is there anything more known about this? In particular, for how long did the increased DNSKEY query rate continue?
-- Chris Thompson Email: cet1@cam.ac.uk _______________________________________________ ksk-rollover mailing list ksk-rollover@icann.org https://mm.icann.org/mailman/listinfo/ksk-rollover
I see the revoked KSK-2010 has gone from the root servers on schedule (as of SOA serial 2019032200). It should be interesting to see whether the DNSKEY query rate to the root servers (1) increases even more (2) falls off now (3) stays much the same. That might give us a whole 1.58 bits of clue as to the cause of the increase in the first place... :-) -- Chris Thompson Email: cet1@cam.ac.uk
On Mar 22, 2019, at 12:54 PM, Chris Thompson <cet1@cam.ac.uk<mailto:cet1@cam.ac.uk>> wrote: I see the revoked KSK-2010 has gone from the root servers on schedule (as of SOA serial 2019032200<tel:2019032200>). It should be interesting to see whether the DNSKEY query rate to the root servers (1) increases even more (2) falls off now (3) stays much the same. That might give us a whole 1.58 bits of clue as to the cause of the increase in the first place... :-) The root zone without KSK-2010 went out ~1440 UTC. We've already seen a massive drop in DNSKEY queries from everywhere we have data. Note the graph below. Matt [cid:5ECC30BA-3FDF-4B27-B657-714849412377]
On 22/03/2019 17:12, Matt Larson wrote:
The root zone without KSK-2010 went out ~1440 UTC. We've already seen a massive drop in DNSKEY queries from everywhere we have data. Note the graph below.
I'm seeing the same on our F-root nodes - a 95% reduction on the volume seen a couple of days ago. Ray
On Mar 22, 2019, at 1:45 PM, Ray Bellis <ray@isc.org> wrote:
On 22/03/2019 17:12, Matt Larson wrote:
The root zone without KSK-2010 went out ~1440 UTC. We've already seen a massive drop in DNSKEY queries from everywhere we have data. Note the graph below.
I'm seeing the same on our F-root nodes - a 95% reduction on the volume seen a couple of days ago.
Do we have an understanding of WHY we are seeing the massive drop in volume? (Or why we saw the massive volume in the first place?) Just curious, Dan -- Dan York Director of Web Strategy, Internet Society york@isoc.org +1-603-439-0024 Jabber: york@jabber.isoc.org Skype: danyork http://twitter.com/danyork http://www.internetsociety.org/
On Mar 22, 2019, at 4:45 PM, Dan York <york@isoc.org> wrote:
On Mar 22, 2019, at 1:45 PM, Ray Bellis <ray@isc.org> wrote:
On 22/03/2019 17:12, Matt Larson wrote:
The root zone without KSK-2010 went out ~1440 UTC. We've already seen a massive drop in DNSKEY queries from everywhere we have data. Note the graph below.
I'm seeing the same on our F-root nodes - a 95% reduction on the volume seen a couple of days ago.
Do we have an understanding of WHY we are seeing the massive drop in volume? (Or why we saw the massive volume in the first place?)
Just curious,
+1
Hi Dan, I found evidence of a software but that may have caused some of the traffic increase. I'm waiting for a bug report to be released as it's being looked into now. Though my guess was it wouldn't drop after the key removal, so I'll stay tuned too! On March 22, 2019 1:45:04 PM PDT, Dan York <york@isoc.org> wrote:
On Mar 22, 2019, at 1:45 PM, Ray Bellis <ray@isc.org> wrote:
On 22/03/2019 17:12, Matt Larson wrote:
The root zone without KSK-2010 went out ~1440 UTC. We've already seen a massive drop in DNSKEY queries from everywhere we have data. Note the graph below.
I'm seeing the same on our F-root nodes - a 95% reduction on the volume seen a couple of days ago.
Do we have an understanding of WHY we are seeing the massive drop in volume? (Or why we saw the massive volume in the first place?)
Just curious, Dan
-- Dan York Director of Web Strategy, Internet Society york@isoc.org +1-603-439-0024 Jabber: york@jabber.isoc.org Skype: danyork http://twitter.com/danyork
http://www.internetsociety.org/ _______________________________________________ ksk-rollover mailing list ksk-rollover@icann.org https://mm.icann.org/mailman/listinfo/ksk-rollover
Wes Hardaker USC/ISI
participants (10)
-
Chris Thompson -
Dan York -
Eric Osterweil -
Geoff Huston -
Matt Larson -
Paul Hoffman -
Ray Bellis -
Roy Arends -
Wes Hardaker -
Wessels, Duane