April 2019 - ksk-rollover - lists.icann.org

Retention of the 2010 KSK
by Kim Davies Sept. 25, 2019

Sept. 25, 2019

Just confirming my mic comments: Our current schedule has us remove the 2010 KSK from our HSMs in one of our two key management facilities in May, and from the HSMs in the other key management facility in August. While perhaps not a complete specification, we’d need a strong indicator we need to retain the KSK longer ideally by May, and certainly no later than August, in order to defer the deletion and retain the capability to use it (i.e. to create a signature via a new mechanism that would endorse the subsequent KSK). kim

13 24

Retention of the 2010 KSK
by David Prangnell April 25, 2019

April 25, 2019

To Whom It May Concern, We have carefully reviewed the recent discussions about retaining KSK-2010 beyond its scheduled lifetime to enable a possible future as-yet-undefined technique to bootstrap a validator that has been offline for an extended period. We have decided to proceed with the deletion of the KSK-2010 as scheduled on 16 May 2019 from the Key Management Facility (KMF) East and then on 14 August 2019 from the KMF West. We have made the decision based on these factors: * On 11 January 2019, the root zone was published with KSK-2010 marked as revoked. The KSK-2010 key was also marked as expired in the root-anchors.xml file. * Since 22 March 2019, the root zone is no longer published with KSK-2010 in the DNSKEY record set. * We have not received a strong indication of how the KSK-2010 would be used in the future. * It seems likely any technique to bootstrap offline validators would be implemented in software that can reasonably assumed to, at a minimum, be configured with KSK-2017. * Deletion of the KSK-2010 is an activity prescribed in the KSK rollover plan [1] and also in the DNSSEC Practice Statements (DPS) [2]. Thank you, -- David Prangnell Email: david.prangnell(a)iana.org<mailto:david.prangnell@iana.org> IANA [1] Page 15 at https://www.icann.org/en/system/files/files/ksk-rollover-operational-implem… [2] Section 6.5 at https://www.iana.org/dnssec/dps/ksk-operator/ksk-dps.txt

4 3

Re: [ksk-rollover] Stand-by KSK for algorithm rollover
by Davey Song(宋林健) April 24, 2019

April 24, 2019

Hi folks, For your information, I setup a testing page for the second lab test of algorithm rollover: https://yeti-dns.org/alg-roll-test.html . The difference between the first trial and the second is that we add stand-by key and have a powerDNS resolver in this testbed. This experiment will start from April 29th and end on 1st of Jun. If you are interested, please join us and follow the instruction on the page. **Note that the experiment will start on April 29th, 0200 UTC when new KEY and signatures will be published. Resolvers setup after that time are not able to roll automatically.** Best regards, Davey 发件人: Davey Song(宋林健) [mailto:ljsong@biigroup.cn] 发送时间: 2019年4月10日 18:32 收件人: 'ksk-rollover(a)icann.org' 主题: Stand-by KSK for algorithm rollover Hi folks, I noticed that no stand-by KSK is pre-published in 2017-ksk rollover, right? I put it due to the limitation of size of DNS response. Any other concerns on stand-by KSK in real production network? Now I’m planning to put a stand-by key in algorithm rollover in my lab test. Because I think ECDSA saves much space than RSA, so maybe it is time to consider Stand-by key for algorithm rollover. Is there any special consideration should be taken care for stand-by key in algorithm rollover. Thanks in advance. Best regards, Davey

1 0

Stand-by KSK for algorithm rollover
by Davey Song(宋林健) April 11, 2019

April 11, 2019

Hi folks, I noticed that no stand-by KSK is pre-published in 2017-ksk rollover, right? I put it due to the limitation of size of DNS response. Any other concerns on stand-by KSK in real production network? Now I’m planning to put a stand-by key in algorithm rollover in my lab test. Because I think ECDSA saves much space than RSA, so maybe it is time to consider Stand-by key for algorithm rollover. Is there any special consideration should be taken care for stand-by key in algorithm rollover. Thanks in advance. Best regards, Davey

3 2

Re: [ksk-rollover] Stand-by KSK for algorithm rollover
by Joe Abley April 10, 2019

April 10, 2019

On 10 Apr 2019, at 06:31, Davey Song(宋林健) <ljsong(a)biigroup.cn> wrote: > I noticed that no stand-by KSK is pre-published in 2017-ksk rollover, right? I put it due to the limitation of size of DNS response. I think it's more to do with the fact that no standby KSK has been generated. Joe

1 0

(Un)planning future KSK replacements
by Salz, Rich April 5, 2019

April 5, 2019

At the BoF at IETF 104 I suggested the following. Make the next KSK rollover scheduled. After that, do not announce them. It is the only way to train the infrastructure to be ready to handle emergencies.

19 37

Description of my analysis of the too-many-KSK queries problem
by Wes Hardaker April 5, 2019

April 5, 2019

TL;DR Background: After the revoke bit was set, all the roots suddenly started receiving significant increases in DNSKEY queries. No one knew why. TL;DR below: I was able to start a bind resolver with specific DNSSEC config settings that caused 6+ outgoing requests for root/DNSKEYs for every incoming request. Then it would go silent, and then it would start over-querying again... ---- In the end, shortly before the revoked key was removed from the published keys list on March 22, I took a quick look into possible causes. I discussed some of my findings, without conclusions, at the DNSSEC workshop [1], as did Duane. After the workshop concluded, I noticed that a bunch of addresses sending lots of queries were from the CS department Purdue. We (ISI) reached out to them asking for help in discovering what the problem was. They discovered that one of the CS labs, which made use of VMs, had left a bunch of VMs running that had DNS resolvers and authoritative servers on them. They were kind enough to send me a copy of the lab setup instructions (which included bind running on Lubuntu), which allowed me to reproduce the situation. The key to the problem turns out that they were specifically using these settings in the bind config file: //dnssec-validation auto; dnssec-enable no; The first link clearly sets the validation state to the default behavior (IE, it was explicitly commented out in the file, and not set to a particular state). I believe the default state is "auto", though I don't know for which versions of bind this is true. The second line, dnssec-enable, was deliberately set to "no". With these two settings adjusted in my default bind.conf file from Fedora (I didn't have Lubuntu to test against), and a bind "managed keys" file that contained only the KSK2010 key and the dlv.isc.org key, I started up bind (bind-9.11.5-4.P4 from Fedora) and tcpdump in parallel. The results were (frustratingly) intermittent. Sometimes there was an issue, and sometimes not. In the end the managed-keys file was updated to contain just the KSK2017 key. When there was an issue, every query I sent from dig@localhost to the resolver@localhost caused 6-7 outgoing queries for DNSKEY and the root servers (I think it required requests for a new TLD each request). After a number of queries, negative caching seemed to finally kick in as the resolver would stop sending DNSKEY queries for a while. Then, on the order of minutes, later it would starting sending more DNSKEY queries in bulk for every incoming request. And as I said, sometimes the resolver failed to even enter the "broken" state. Evan, at the IETF, reported in a few meetings and conversations that they had discovered a bug in bind previously that would exhibit this roll-over-and-die type behavior but that it was only present in out-of-date versions of bind (9.10 and below I believe he stated). I appreciate the bind team looking into the problem. I can't, unfortunately, reconcile why my bind 9.11 on Fedora seemed to have the same issue however. My colleague Robert Story reproduced the issue as well and had tried Fedora 27-29, though I don't remember which versions had the issue. [1]: https://icann.zoom.us/recording/play/OYJ4R7IQCxnF5Rw6Kk_nnJQqrp4037W7eS9eeJ… -- Wes Hardaker USC/ISI

3 6

Bind sending too many DNSKEY queries today, even in recent versions
by Wes Hardaker April 4, 2019

April 4, 2019

I couldn't figure out why a caching bug would likely be restricted to just the period where the key was published with a revoke bit set. So I decided to re-try the experiment I did in the airport, but this time with data captures and "real science". The results contain graphs, so rather than write it up here, I'll direct you over to my write up on this web site: https://www.isi.edu/~hardaker/news/20190404-bind-bug.html TL;DR: some versions of bind (at least), in certain configurations, still sometimes (15% of the time) enter a state of sending a large number of DNSKEY queries on a regular basis. Today. -- Wes Hardaker USC/ISI

1 0

Why keep old private keys?
by Salz, Rich April 4, 2019

April 4, 2019

What is the point of keeping old keys around? So we can sign “something” in case an old DNS server gets turned back on? What is the downside? Leaving old key material around that could be stolen. (Is someone going to count the HSM’s every time a safe is opened, for example?) We’re trading ease of resurrecting old machines against the security of the DNSSEC world. Like Mike said: DO NOT KEEP OLD KEYS. If you think you’ll need something signed, then sign the new key and then destroy the HSM.

5 5

Thoughts on future KSK Rolls
by Tim April April 4, 2019

April 4, 2019

Below is the distillation of my notes regarding the future of KSK rolls. Without a doubt, many of the ideas below have come from conversations with others or statements made on the list or in the meetings, but this is my current view of things. --tim Rotation Period: The key should be rotated on a regular schedule. What a "regular schedule” is up for debate, probably with parties from the ICANN Community, ICANN org (likely the Office of the CTO - OCTO) and IETF participants (likely interested parties from dnsop). The starting point for the conversation would be to consider the guidance set out in the [DNSSEC Practice Statement for the Root Zone KSK Operator] document, section 6.5. (Sidenote: I believe that section 6.5 should have been worded differently to be more clear on the intentions of the authors, possibly saying "Each RZ KSK will be scheduled to be rolled over through a key ceremony as required, and to happen at least once every five years”). The rotation period of five years, in my opinion, is far too long and can result in operators and developers becoming complacent, or being generally unaware, that keys rolls happen. My current opinion is that the rotation period should be roughly once a year. The yearly cadence would allow the key to be somewhat stable but also have the key rolls be a regular event that can be predicted while not falling too far out of working memory. In thinking about rotations, I started to wonder if the timing for the whole process should be on a predictable, fixed day of week and time of year as to be predictable, even before formal timelines are published. The scheduling of the critical dates (like changes to the root zone file) should probably be scheduled in the same way that the US holidays of Thanksgiving, Memorial Day, Labor Day or Election Day is done; where a specific day of the week and a specific week in the year are specified (eg 4th Thursday in November for US Thanksgiving). A proposed schedule for the process could be like the following: --- Process Start --- 1. Key Generation (Site A): First Tuesday of September 2. Key Replicated to Alternate Site: As able but before the next step 3. Key Published in IANA: First Tuesday in October 4. Key Published in the Root Zone: First Tuesday in November 1. ( allow 2 months for RFC5011 to work ) 5. New KSK switch to sign the root zone: Second Tuesday in January 1. ( pushed out a week to allow for New Years Celebrations ) 2. ( allow 2 months before revocation ) 6. Old Key Revoked Bit Set: Second Tuesday in March 1. ( allow 2 months before removal ) 7. Old Key removed from Root Zone: Second Tuesday in May --- Process Complete --- I have not been very rigorous about the dates suggested above, but something similar to the [IETF Clash List] should be considered at least at the start of the roll process to try and limit the collateral damage that might be caused by an accident while the process starts up. It would also be wise to consider major Holidays like Passover and Christmas as well. When thinking about the schedule above the following considerations were made: * avoid US major / bank holidays on key steps * avoid Patch Tuesday during the first half of the process until the new key is published * skip Mondays (holidays tend to fall on Mondays). This also helps avoid people who take long weekends. * skip Fridays / Weekends: many senior engineering / ops staff have weekends off. Also changing things on Friday tends to have unnoticed failures until the following week. Each of the steps listed above should also have some sort of communication around it, ideally with some pre-notification and then a confirmation message following the completion. When the rolls start to become more commonplace, the number of notifications may start to go away, but I think that would be community driven as we get better at rolling the KSK. Key Management Methodology The current steady state of the KSK has one valid KSK. To this point, this has not been a problem, but this also means that if an emergency keyroll is required, the current tooling / support will not help with a roll. A potential way to support an emergency key roll, should one be needed, would be to have a backup key created and staged in the root zone. To take this approach, I would propose a model where, outside of an emergency condition, there would be at least two KSKs in the root zone at any time which have been published for at least one month (the 5011 hold down timer length). To achieve this, using the schedule proposed above, the steps 1-4 would happen, creating a new key. Step 5 would promote a key that has been in the root zone for 14 months to active and then step 6 would be on a key that has been provisioned for 26 months before it is then removed from the zone two months later. For a six-month window, there would end up being three keys in the root zone, but one would be a hot standby, ready for use in the event of an emergency. Here is an example of the key rolling broken down State 1: KSKA (signing), KSKB (next key / emergency key) State 2: KSKA (signing), KSKB (next key / emergency key), KSKC (new) State 3: KSKA, KSKB (signing), KSKC (next key / emergency key) State 4: KSKA (revoked), KSKB (signing), KSKC (next key / emergency key) State 4: KSKB (signing), KSKC (next key / emergency key) Having three keys (using the algorithm currently in use by the root zone, RSA/SHA-256) would likely make the DNSKEY query response quite large resulting in increased traffic to the root servers. Before considering this model, I would defer to OCTO and the RSSAC for an assessment of the state of the root servers. To support an expanded number of keys in the root, a key algorithm roll might also be an interesting conversation area. Having now talked about an algorithm roll, I also want to bring up another possible concern when it comes to cryptographic keys: potential weaknesses in algorithms. It may be worth considering a backup key that is a different algorithm family from the other keys that are provisioned in the event that an algorithm is needed expeditiously. Considerations of KSK2010 I’ve seen the discussion that happened in Prague at IETF104 about the scheduled destruction of KSK2010. I’m not convinced it is the best idea to destroy the key yet assuming it is still stored in the same way as it was when it was the active key. To Wes’ point during the KSK BoF, it might be interesting to find some way to chain all KSKs that are in use back to a KSK, such as KSK2010, but then there is the issue that code that still relies on KSK2010 will not have any new method to bootstrap up from KSK2010, so the point might be moot. I think it would be wise to consider other options of things we might want to do with that key before it goes away, but to also keep in mind that we might want to just treat KSK2010 as dead and pick up on KSK2017 (or some successor) as the initial trust anchor. IETF Clash List: http://www6.ietf.org/meeting/clash-list.html DNSSEC Practice Statement for the Root Zone KSK Operator: https://www.iana.org/dnssec/icann-dps.txt

5 8