March 2019 - ksk-rollover - lists.icann.org

Retention of the 2010 KSK
by Kim Davies Sept. 25, 2019

Sept. 25, 2019

Just confirming my mic comments: Our current schedule has us remove the 2010 KSK from our HSMs in one of our two key management facilities in May, and from the HSMs in the other key management facility in August. While perhaps not a complete specification, we’d need a strong indicator we need to retain the KSK longer ideally by May, and certainly no later than August, in order to defer the deletion and retain the capability to use it (i.e. to create a signature via a new mechanism that would endorse the subsequent KSK). kim

13 24

(Un)planning future KSK replacements
by Salz, Rich April 5, 2019

April 5, 2019

At the BoF at IETF 104 I suggested the following. Make the next KSK rollover scheduled. After that, do not announce them. It is the only way to train the infrastructure to be ready to handle emergencies.

19 37

Thoughts on future KSK Rolls
by Tim April April 4, 2019

April 4, 2019

Below is the distillation of my notes regarding the future of KSK rolls. Without a doubt, many of the ideas below have come from conversations with others or statements made on the list or in the meetings, but this is my current view of things. --tim Rotation Period: The key should be rotated on a regular schedule. What a "regular schedule” is up for debate, probably with parties from the ICANN Community, ICANN org (likely the Office of the CTO - OCTO) and IETF participants (likely interested parties from dnsop). The starting point for the conversation would be to consider the guidance set out in the [DNSSEC Practice Statement for the Root Zone KSK Operator] document, section 6.5. (Sidenote: I believe that section 6.5 should have been worded differently to be more clear on the intentions of the authors, possibly saying "Each RZ KSK will be scheduled to be rolled over through a key ceremony as required, and to happen at least once every five years”). The rotation period of five years, in my opinion, is far too long and can result in operators and developers becoming complacent, or being generally unaware, that keys rolls happen. My current opinion is that the rotation period should be roughly once a year. The yearly cadence would allow the key to be somewhat stable but also have the key rolls be a regular event that can be predicted while not falling too far out of working memory. In thinking about rotations, I started to wonder if the timing for the whole process should be on a predictable, fixed day of week and time of year as to be predictable, even before formal timelines are published. The scheduling of the critical dates (like changes to the root zone file) should probably be scheduled in the same way that the US holidays of Thanksgiving, Memorial Day, Labor Day or Election Day is done; where a specific day of the week and a specific week in the year are specified (eg 4th Thursday in November for US Thanksgiving). A proposed schedule for the process could be like the following: --- Process Start --- 1. Key Generation (Site A): First Tuesday of September 2. Key Replicated to Alternate Site: As able but before the next step 3. Key Published in IANA: First Tuesday in October 4. Key Published in the Root Zone: First Tuesday in November 1. ( allow 2 months for RFC5011 to work ) 5. New KSK switch to sign the root zone: Second Tuesday in January 1. ( pushed out a week to allow for New Years Celebrations ) 2. ( allow 2 months before revocation ) 6. Old Key Revoked Bit Set: Second Tuesday in March 1. ( allow 2 months before removal ) 7. Old Key removed from Root Zone: Second Tuesday in May --- Process Complete --- I have not been very rigorous about the dates suggested above, but something similar to the [IETF Clash List] should be considered at least at the start of the roll process to try and limit the collateral damage that might be caused by an accident while the process starts up. It would also be wise to consider major Holidays like Passover and Christmas as well. When thinking about the schedule above the following considerations were made: * avoid US major / bank holidays on key steps * avoid Patch Tuesday during the first half of the process until the new key is published * skip Mondays (holidays tend to fall on Mondays). This also helps avoid people who take long weekends. * skip Fridays / Weekends: many senior engineering / ops staff have weekends off. Also changing things on Friday tends to have unnoticed failures until the following week. Each of the steps listed above should also have some sort of communication around it, ideally with some pre-notification and then a confirmation message following the completion. When the rolls start to become more commonplace, the number of notifications may start to go away, but I think that would be community driven as we get better at rolling the KSK. Key Management Methodology The current steady state of the KSK has one valid KSK. To this point, this has not been a problem, but this also means that if an emergency keyroll is required, the current tooling / support will not help with a roll. A potential way to support an emergency key roll, should one be needed, would be to have a backup key created and staged in the root zone. To take this approach, I would propose a model where, outside of an emergency condition, there would be at least two KSKs in the root zone at any time which have been published for at least one month (the 5011 hold down timer length). To achieve this, using the schedule proposed above, the steps 1-4 would happen, creating a new key. Step 5 would promote a key that has been in the root zone for 14 months to active and then step 6 would be on a key that has been provisioned for 26 months before it is then removed from the zone two months later. For a six-month window, there would end up being three keys in the root zone, but one would be a hot standby, ready for use in the event of an emergency. Here is an example of the key rolling broken down State 1: KSKA (signing), KSKB (next key / emergency key) State 2: KSKA (signing), KSKB (next key / emergency key), KSKC (new) State 3: KSKA, KSKB (signing), KSKC (next key / emergency key) State 4: KSKA (revoked), KSKB (signing), KSKC (next key / emergency key) State 4: KSKB (signing), KSKC (next key / emergency key) Having three keys (using the algorithm currently in use by the root zone, RSA/SHA-256) would likely make the DNSKEY query response quite large resulting in increased traffic to the root servers. Before considering this model, I would defer to OCTO and the RSSAC for an assessment of the state of the root servers. To support an expanded number of keys in the root, a key algorithm roll might also be an interesting conversation area. Having now talked about an algorithm roll, I also want to bring up another possible concern when it comes to cryptographic keys: potential weaknesses in algorithms. It may be worth considering a backup key that is a different algorithm family from the other keys that are provisioned in the event that an algorithm is needed expeditiously. Considerations of KSK2010 I’ve seen the discussion that happened in Prague at IETF104 about the scheduled destruction of KSK2010. I’m not convinced it is the best idea to destroy the key yet assuming it is still stored in the same way as it was when it was the active key. To Wes’ point during the KSK BoF, it might be interesting to find some way to chain all KSKs that are in use back to a KSK, such as KSK2010, but then there is the issue that code that still relies on KSK2010 will not have any new method to bootstrap up from KSK2010, so the point might be moot. I think it would be wise to consider other options of things we might want to do with that key before it goes away, but to also keep in mind that we might want to just treat KSK2010 as dead and pick up on KSK2017 (or some successor) as the initial trust anchor. IETF Clash List: http://www6.ietf.org/meeting/clash-list.html DNSSEC Practice Statement for the Root Zone KSK Operator: https://www.iana.org/dnssec/icann-dps.txt

5 8

new ksk and DNS software vendors
by manu tman March 30, 2019

March 30, 2019

Hi all, During the BoF session this morning, it was asked how long it would take vendors to incorporate the new KSK in their software. The few that spoke said it was a relatively short time. This is fine for people that get the latest versions and install it, but involves more communication between multiple parties when the goal is to update the keys and does not involve binary change. The main issue with going through that route is that the new key is only useful once it makes it into the distros that ship those softwares which involves DNS software vendors to work with upstream distros to get it in. One approach I would suggest is to rather work with DNS vendors to make sure they can all read the keys from a given format(s) (which I am sure is already the case) and then work with distros to make sure that all the DNS software they ship uses the same file. This file can then be distributed via a `trust-anchor` package à la `ca-certificates` for RedHat and Debian based distros. There is obviously an existing process for that, so I am hopeful it could be replicated for getting trust anchors from IANA. Automation on the distro side to pick up new trust anchors also seem rather trivial. I would love to hear from people closer to the distros realm if this is not, but it seems something that could be quite easily addressed and would be sustainable long term. The update will apply uniformly to all DNS softwares shipped by the distros, there is no need to rebuild/recompile anything which involves DNS softwares, the package is pretty trivial to update and assuming 4 major OSS DNS softwares, overhead drops by 3/4, or even close to 0 if some form of automation is put in place. Manu

5 6

KSK2017 Rollover was a success
by Ólafur Guðmundsson March 30, 2019

March 30, 2019

Repeat from what I said at the microphone today Main lesson from this roll is it worked better than we could have expected, given this was the first time, We expect that software/configuration has bugs/errors and this exposed some. There might have been some configurations that did not anticipate change in the key used ==> nothing beside rolling the KSK could have exposed that. There were some outages, there may have been some sites that turned off DNSSEC and we need to get some measurements of what that long term effect was i.e. did the validation get turned back on. The traffic increase reported was interesting but the big picture is it was in the NOISE range, i.e. all root servers should be able to deal with such small increase. I have no opinion at this point when next to roll or how fast to perform that roll. Ólafur

4 3

On the subject of timing
by Matthew Pounsett March 29, 2019

March 29, 2019

Repeating (with a little more care and hopefully clarity) my point from the mic the other morning.. Now that we've got a roll under our belts, I think it's time to move to a condition where the choice of how often to roll the root KSK is mostly a matter of an operational choice of the Root Zone Partners (noting the formal meaning of that term). I would suggest that, as one of the goals of regular rolls is operational practice, the rate of rotation should not be too slow; I think what that means, exactly, is up to the RZP, however. Also keeping in mind that there are, very likely, people in the community that disagree with me on that subject, the RZP should put the timing of the next roll out for public comment (as has been done in the past) but after some sort of rough consensus on the next roll date is reached, make it a matter of operational choice from that point forward. With that distraction out of the way of the community at large, we can get on with the business of figuring out how to do emergency key rolls, bootstrap old hardware, and solve other problems not covered by 5011. All things I shall surely comment on in other threads.

1 0

FYI - notes from KSK Future BoF at IETF 104
by Dan York March 29, 2019

March 29, 2019

FYI, I and others took some notes of the conversation in the KSK Future BOF this week at IETF 104 in Prague: https://etherpad.ietf.org/p/notes-ietf-104-ksk?useMonospaceFont=true Some of the people who commented have sent their own comments directly to this list (and I would suggest reading *those* comments versus the summaries that were made in the notes). Dan -- Dan York Director of Web Strategy, Internet Society york(a)isoc.org +1-603-439-0024 Jabber: york(a)jabber.isoc.org Skype: danyork http://twitter.com/danyork http://www.internetsociety.org/

1 0

Gathering operators' voice
by Yoshiro YONEYA March 28, 2019

March 28, 2019

Following is what I said during KSK Future BoF today. To gather operators' voice widely, conducting survey is a way. Operators are shy to express their experiences to the public mailing list, but they can answer anonymous survey, and then surveyer can proxy their answers anonymily. We can do it in our local community in local language, and bring it back to this mailing list. -- Yoshiro YONEYA

3 2

suggestions for deciding on key roll timing
by Wes Hardaker March 28, 2019

March 28, 2019

I just mentioned this at a mic, and I'm re-broadcasting it here: I believe we should consider two different elements separately for deciding future key-roll strategies: 1) When we should perform the next roll 2) What frequency we should use between future rolls after the next And for #1, I believe we shouldn't be thinking about time at all, but rather we should create two lists of MUSTs and MAYs for rolling next, and ensures all MUSTs are handled. The MUST and MAY list should include both "do this" type entries, as well as "think about this (again)" type entries. After thinking about the later "think about" items, they may, of course, turn into "do this" type entries. tl;dr: we need a list of blockers that must be dealt with before rolling. -- Wes Hardaker USC/ISI

4 3

Need to roll regularly to ensure operational stability.
by Lars-Johan Liman March 28, 2019

March 28, 2019

All, As I stated at the mic in the BOF: We should look at _why_ we want to roll the key. My take is that we want to do this to make sure that rolling the key is a non-event. We need to make sure that all resolvers handle this in a smooth fashion, and that all operators remember what to do when it happens. Ideally it will be automated everywhere. My analogy was the diesel generators in a data centre. You don't wait for The Big Power Outage to run them, you do it every month, so that you discover potential problems, and so that all staff and equipment are "trained" to act appropriately. Repeating the event is the only way to train the world that this is a non-event. Make the "pain" is frequent enough, and people will make sure they handle it appropriately to avoid the pain. --> Automation. With a smooth machinery, we can roll at any time, should an emergency arise. And yes, there is a non-trivial cost associated with rolling, but again, we can only drive down the cost by going for "mass-production". The more often we do it, the lower the cost must be, and will eventually become. I suggest we roll on a yearly basis, but I can be convinced to agree to every two years. I also kind of like the idea to perform unannounced rolls in the future. It. Must. Be. A. Non-event! Cheers, /Liman -- #---------------------------------------------------------------------- # Lars-Johan Liman, M.Sc. ! E-mail: liman(a)netnod.se # Senior Systems Specialist ! Tel: +46 8 - 562 860 12 # Netnod Internet Exchange, Stockholm ! http://www.netnod.se/ #----------------------------------------------------------------------

2 1