- ksk-rollover - lists.icann.org

Suggested update to the key ceremonies.
by Warren Kumari Feb. 21, 2018

Feb. 21, 2018

Apologies if this isn't the right place to propose this - the ksk-ceremony list didn't feel right... I think that it would be a useful addition to the script to ensure that, when a new KSK is generated, it does not have the same Key ID as any previous KSKs. It is *does* have the same Key ID, it should be discarded and a new one generated. Rational: If we end up with multiple keys with the same Key ID it becomes very tricky to run things like RFC8145, KSK Sentinel, etc. Also, when doing troubleshooting / diagnostics, the key ID is an easy thing to use to differentiate keys. This has long been source of low level concern for me, and I've been assured that if there were collisions during the ceremony, the right thing would likely happen -- but I think that this is worth explicitly noting what happens. I *did* look at the scripts, and didn't see a note on this; 'pologies if it is already covered and I missed it. W -- I don't think the execution is relevant when it was obviously a bad idea in the first place. This is like putting rabid weasels in your pants, and later expressing regret at having chosen those particular rabid weasels and that pair of pants. ---maf

15 22

Concerns about KSK generation software quality.
by Warren Kumari Feb. 16, 2018

Feb. 16, 2018

In the "Suggested update" thread, Andres pointed me (http://mm.icann.org/pipermail/ksk-rollover/2018-February/000385.html) at the IANA DNSSEC-Keytools KSK generation tools - https://github.com/iana-org/dnssec-keytools/blob/master/kskgen/kskgen.c I took a quick look at the code, and much of it scares me. I suspect that Andres was talking about the pkcs11 CKA_LABEL label, which turns into a "keypair label" - from the code, this is unique because (dnssec-keytools/common/pkcs11_dnssec.c): /* * CKA_LABEL HACK * AEP Keyper can only display 7 characters and * cannot change the HSM internal CKA_LABEL once created. * So, we label them with a monotonically increasing * string based on seconds since epoch. */ This is (AFAICT), simply an internal label for the key, but is distinct / unrelated to the DNSSEC keytag (as far as I can see). The DNSSEC key tag is generated (by dnssec_keytag in dnssec-keytools/common/dnssec.c:25) purely from the [Flags, Prot, Alg, PubKey] (RFC4034, Appendix B). The comments interested me, so I looked a little bit further, and now I'm filled with unease. What is concerning to me is that the "void *pkcs11_genrsakey(int bits,int flags)" function (dnssec-keytools/common/pkcs11_dnssec.c:1121) tries 10 times to find a unique keypair label, but if it cannot, throws an error about DNSSEC keytags: if(ts > 0) { /* DUPLICATE - try again */ logger_warning("Found %u duplicate keys labeled \"%s\". Try again...",ts,label); if(++tries < 10) { sleep(1); goto regen; } logger_error("Can't get a unique DNSSEC tag after %d tries. Giving up...",tries); goto err; } Note that this all happens before the actual key generation (~1248). While this is obviously just a misleading error message (it's not a DNSSEC tag, it is a key label), it makes me wonder about how carefully the whole CKA_LABEL HACK bit has been reviewed. Looking a bit further, the keylabel comes from get_monotonic_str() (dnssec-keytools/common/pkcs11_dnssec.c:1087). This bit of code seems, er, interesting. It starts with: static time_t t1=0; I'm surprised by the "static time_t" - function-local static variables lead to all sorts to hard to debug side effects. Anyway, I also cannot understand how the output of this could cause duplicates, and so don't understand why this is being tried 10 times - if there **is** already a keypair with this label, surely this is evidence of much larger issue, and simply trying again is not a good idea? A quick look also shows a bunch (13) of gotos, including at least 3 instances of "goto err;" without any useful sort of log message being printed (e.g: if((rv=pfl->C_FindObjects(sh,hKeys,PKCS11_MAX_KEYS_PER_SLOT,&ts)) != CKR_OK) goto err; -- this will fail with no output as to why.) A *very* quick skim of the source code also shows all sorts of typoes: if(t1) { /* since we are dropping the last base32 char, wait to gurantee uniqueness */ --- guarantee The CKA_LABEL is based on a time based sting for uniquness. --- string, uniqueness logger_error("Key generaton failed"); --- generation * create a CSR - a simple private key proof of possesion ---- possession \return NULL if failed; otherwise newlly crated key record associated with pk. --- newly created /* setup computation for DS rcords */ --- records /*! check to see if we have access to the private key in this key strust --- struct? /*! log in to HSM slot referrenced by pk --- referenced /* get correspoding private key if possible */ --- corresponding While typos in error messages and comments are (clearly) not going to cause breakage, it does make me concerned that this code, which generates the single most critical DNSSEC keys, has not received sufficient careful review. These sorts of obvious typos are not themselves a problem, but rather indicative of a larger issue. While the obvious retort is "It is publicly posted in GitHub, and we asked the community to have a look. Send pull requests!", it appears that this hasn't resulted in enough review. I used to program C, but it has been long enough that now I'm only qualified to say "Well, that doesn't seem right!" - how do we get (more) review from people who are better qualified? The KSK ceremonies provide important transparency - how can we get more review of the tools used in the ceremonies? Having people fly in, and then having the tools fail, or, worse yet, questions about the correctness of the result does not seem good. W -- I don't think the execution is relevant when it was obviously a bad idea in the first place. This is like putting rabid weasels in your pants, and later expressing regret at having chosen those particular rabid weasels and that pair of pants. ---maf

2 1

Graph showing latest RFC8145 data
by Matt Larson Feb. 8, 2018

Feb. 8, 2018

Dear colleagues, In the Plan for Continuing the Root KSK Rollover,<https://www.icann.org/en/system/files/files/plan-continuing-root-ksk-rollov…> we wrote: ICANN org will publish monthly snapshots of the RFC 8145 trust anchor report data that we receive from several root servers. Our plan is to make daily updating graphs available on a web page Real Soon Now. In the meantime, I wanted to share the latest graph of RFC8145 trust anchor report data. We're receiving data from nine root letters (A, B, C, D, F, J, K, L and M), though the list of letters contributing has grown over time: B, D and F from 2017-09-01 A/J added on 2017-12-22 H added on 2018-01-03 C added on 2018-01-04 L and M added on 2018-01-10 (Thank you very much to those root operators for contributing the data!) Presumably these additions over time account for some of the jumps in the graph, but we've not (yet) made an extensive study of that. However, we do not believe the largest increase is attributable to more root letters contributing data: the recent increases are presumably due to the release of Unbound 1.6.8 on 19 January 2018, which is a bug fix for CVE-2017-15105 (a vulnerability in the processing of wildcard synthesized NSEC records). While trust anchor signaling has been on by default in Unbound since version 1.6.7, the need to address the vulnerability has prompted a more rapid upgrade to 1.6.8 and a corresponding increase in trust anchor reports. Burying the lede, you'll note the significant increase in the percentage of sources reporting only KSK-2010. We haven't investigated that yet, either, but we certainly will. Matt

1 0

Re: [ksk-rollover] [Ext] Re: Starting discussion on acceptable criteria for proceeding with the root KSK roll
by wfms＠wfms.org Feb. 6, 2018

Feb. 6, 2018

Hi Paul, Matt and all. (Switching hats) I've followed the discussion quite a bit on the mailing list, and feel that pretty much most of what needed to be said was said, so I'll leave at that and not repeat anything. But here are 2c: I admire the fact that ICANN are taking this on the chin, delaying the roll-over in deference to the end user. However, I feel ICANN may be overstepping the mark: Ultimately, it is up to the operators to 'opt-in' and use DNSSEC. Aside from the TLDs you have a say over, ICANN don't have the authority (and to a degree, the responsibility) to dictate to these resolver operators: No gun was ever held to their heads to force them to adopt DNSSEC. That said, I disagree with a number here who say 'just do it.' There be dragons and I agree with Geoff, there's the issue of inviting comparisions to the Wild West of the Internet. I would offer this: Announce a tentative date of April 11, 2018 and see who pipes-up (just tell them I did it). Operators (negligent, diligent, absent) can only be warned for so long and so much that ultimately the warnings fall on deaf ears or people just get jaundiced and don't believe it will happen. The danger of setting a date is to have it moved again and losing further operational trust. Those suggesting to wait for more data (if there is a reasonable expectation of more coming) I could suggest a date of October 11, 2018. It gives time to determine if a trend can be established plus it gives a unique opportunity of a longer-term transition by having KSK 2010 and KSK 2017 hang around longer together and ultimately studying the effect of that too. Plus it gives more time for operators to actually fix things. In summing and to bring this back to the original poster's question, here's a stab at criteria for setting a date for the complete rollover (in no particular order): 1) measuring the value of trust in DNSSEC 2) having the best available data 3) a trend showing an uptake of KSK 2017 reaching the best rate approaching a flatline in that data. If a date can be agreed to, what happens next is to do another reach out, but more targetted. For example, for those registrars that offer up DS key registration, can they link to a notice from the dialog boxes where they get entered, or publish a one-line blurb about the impending roll-over and what that means? Alternatively here's perhaps a more risky proposition: A combination of 'just do it' but with a twist. Pull the old KSK for a fixed time period (hours?) WITH lot's of prior advance warning on a certain date (someone WILL be collecting data....right?). Long have there been calls by groups to attempt to disrupt the DNS, what's a planned *potential* outage affecting DNSSEC? Seriously though, it is akin to jerking a sleeping dog's chain, but no one really knows if it's a Chihuahua or a Great Dane on the other side of that fence until a hardy pull gets made. I think your note below on communications hits the mark so I'll leave it at that. Thanks, On Wed, 17 Jan 2018, Matt Larson wrote: > > On Jan 17, 2018, at 1:19 PM, Warren Kumari <warren(a)kumari.net> wrote: > > I ment to include the below in my original bloviation: > I think it would be really useful to reach out to the press who published > articles on the > keyroll pause (e.g: BleepingComputer, Bloomberg, Modern Ghana, The Register, > ITWorld, etc) - > having them be told ahead of time that ICANN stopped things, got community > feedback and is > proceeding cautiously (potentially) changes the narrative completely - and, > at least, helps > prevent the bad PR hit to ICANN (this is an ICANN list, after all) and them > feeling > blindsided. Converting the potential PR ding into a win would be nice - and > may also reach > more people. > > > This is a good suggestion and I will add it to the hopper of PR ideas. Please > recall that in late > December > (https://www.icann.org/news/blog/update-on-the-root-ksk-rollover-project) we > wrote: > > The ICANN org will monitor this mailing list and beginning on 15 > January 2018, we will > develop a draft plan for proceeding with the root KSK roll based on the > input received > and discussion on the mailing list. The plan will be published by 31 > January 2018 and > undergo a formal ICANN public comment process to gather further input. > > > We are indeed planning to publish a draft plan for moving forward at the end > of the month based on > this discussion, and we're also planning PR activities to publicize the plan > and the formal public > comment, including outreach to publications that have previously covered the > root KSK roll or that > we suspect would be willing to cover it. > > Matt > >

2 1

Courtesy advance notification: Announcing Draft Plan for Continuing With the Root KSK Roll
by Matt Larson Feb. 5, 2018

Feb. 5, 2018

Dear colleagues, Thank you for your feedback on criteria for continuing with the root KSK roll. Based on the discussion here, ICANN has made a plan for continuing with the root KSK roll and today we are announcing that plan, which is published here: https://www.icann.org/en/system/files/files/plan-continuing-root-ksk-rollov… We're also opening a public comment period to receive wider community feedback. Below is the text of a blog post that will appear on icann.org<http://icann.org> at around 0900 PST. Thanks and please let me know if you have any questions. Matt Announcing Draft Plan For Continuing With The KSK Roll By Matt Larson, VP of Research, Office of Chief Technology Officer A formal ICANN public comment period has been opened to receive community input on a draft plan<https://www.icann.org/en/system/files/files/plan-continuing-root-ksk-rollov…> to proceed with the KSK rollover project. This comment period will run until 1 April 2018 and we are eager to receive any and all comments. The plan calls for rolling the root zone KSK on 11 October 2018 (one year later than originally planned), continuing extensive outreach to notify as many resolver operators as possible, and publishing more observations of the RFC 8145 trust anchor report data. Additional details are contained within the plan. In addition, we are planning a session at ICANN61 in Puerto Rico, to further discuss the plan and obtain additional feedback. The draft plan follows our posting<https://www.icann.org/news/blog/update-on-the-root-ksk-rollover-project> in late December, in which the ICANN organization announced next steps in the process to resume the root KSK rollover project. We described our efforts to track down the operators of DNS resolvers that were not ready for the rollover. Using a protocol described in RFC 8145<https://tools.ietf.org/rfc/rfc8145.txt>, these problematic resolvers had reported to the root servers a trust anchor configuration with only the current KSK (known as KSK-2010) and not the newer KSK (known as KSK-2017). In our December posting we also detailed the difficulty in contacting operators, and noted that when we were able to reach an operator, we learned that there were a variety of causes for the resolver’s lagging configuration. The bottom line is that these findings did not afford much clarity as to the next steps for mitigating specific causes nor did they afford any guidance for appropriate messaging. Faced with this situation, we announced our intention to solicit input from the community on acceptable criteria for proceeding with the root KSK roll. Since that posting in December, a robust community discussion ensued between interested community members. There was agreement during these discussions that there is no way to accurately measure the number of users who would be affected by rolling the root KSK, even though there was a belief that better measurements may become available for future KSK rollovers. The consensus of those involved in the discussions was that the ICANN org should proceed with rolling the root zone KSK in a timely fashion while continuing outreach to ensure that the word of the rollover reach as wide an audience as possible. We look forward to continuing to work with the ICANN community to roll the root zone KSK.

4 6

Re: [ksk-rollover] Architectural reconsideration on ICANN's Root Zone KSK rollover
by Davey Song(宋林健) Feb. 5, 2018

Feb. 5, 2018

Hi S Moonesamy, Thanks for your review and comments. > There is the following in the proposal: "It require new DNS extension > and functions in both authoritative and recursive side servers". How > long would it take to deploy the new DNS extension? Yes. It is an issue I raised in the conclusion part. It may take a long time for obsolete resolvers, but it can be very quick for resolvers willing to cooperate. The proposal is obvious different from ICANN's plan. I'm not in favor of a hurry rollover once for all resolvers. I proposed an phased plan to different resolvers. My argument is that there is two risks: one is old key risk (compromised key) which IMHO is less than the second risk of key rollover for unready resolver. The former one is theoretical and imaginary "enemy". But the latter one is real suffering and the risk is inestimable because we do not know the proportion of unready resolver and their users. So compared the two risks, I propose to roll the key for ready resolver and keep the unready resolver the same. Unfortunately, ICANN and most of people here seems to be in a hurry to roll the key for all resolvers. Sorry my proposal is not for their taste. So I will keep my opinion on this. Best regards, Davey

1 0

Running code: draft-ietf-dnsop-kskroll-sentinel-00 in Knot Resolver 2.0.0
by Petr Špaček Feb. 1, 2018

Feb. 1, 2018

Hello, draft-ietf-dnsop-kskroll-sentinel-00 is now implemented in Knot Resolver version 2.0.0 [1] which was released yesterday, and it is enabled by default. To make things more interesting, version 2.0.0 also has implementation of RFC 8198 Aggressive Use of DNSSEC-Validated Cache, which effectively means that RFC 8145 signaling queries sent by something using our resolver are not going to reach root because they will be blocked by the aggressive cache. Oh well. As I said earlier, I think we are not going to have reliable data in upcomming years, so let's generate some PR and treat KSK-2017 roll as one of many security issues - it will be fixed like any other security issue. [1] Knot Resolver https://www.knot-resolver.cz/ https://www.knot-resolver.cz/2018-01-31-knot-resolver-2.0.0.html -- Petr Špaček @ CZ.NIC

1 0

Architectural reconsideration on ICANN's Root Zone KSK rollover
by Davey Song(宋林健) Jan. 31, 2018

Jan. 31, 2018

Hi folks, I followed the discussion in this ML and composed a draft proposal (attached and in my repo) as an input. I will not proceed in IETF but consider to publish it in other form, if you think it is too radical to implement. Any comments on the proposal or edits for my poooor English is welcome. J The link of my repo: https://github.com/songlinjian/Parallel-Root-KSK-Rollover Best regards, Davey

2 2

Starting discussion on acceptable criteria for proceeding with the root KSK roll
by Paul Hoffman Jan. 17, 2018

Jan. 17, 2018

Greetings in the new year. As announced on this list (and in many other places) a few weeks ago, the ICANN org wants to use this list to get input from the community on acceptable criteria for proceeding with the root KSK roll. When we made that announcement, we saw a good number of new subscriptions to the list, but the discussion didn't start on its own, so we want to get that going. For reference, please see <https://www.icann.org/news/blog/update-on-the-root-ksk-rollover-project>. The relevant timing part from that article is: > The ICANN org will monitor this mailing list and beginning on 15 January 2018, we will develop a draft plan for proceeding with the root KSK roll based on the input received and discussion on the mailing list. The plan will be published by 31 January 2018 and undergo a formal ICANN public comment process to gather further input. We would really like to hear from you about the criteria you think would be relevant for us to observe/measure, if such criteria exist. --Paul Hoffman

19 42

Re: [ksk-rollover] Starting discussion on acceptable criteria for proceeding with the root KSK roll
by S Moonesamy Jan. 14, 2018

Jan. 14, 2018

Hi Geoff, At 04:19 PM 04-01-2018, Geoff Huston wrote: >Carlos, (I'm asking because you posted a "me >too") what is the data set you are using to >justify this call to be "over soon"? It seems >to me that in the absence of new data, the only >changed factor is your own appetite for risk. >Without additional data, your tolerance for risk >appears to increase over time (*). But is this >altered personal perception of the risk >sufficient motivation to proceed? Objectively, >if the numbers in September 2017 gave sufficient >grounds to pause, and the numbers haven't >changed (**) then surely the grounds for pausing >the operation as as strong now as they were in September (***). There is the following in the KSK rollover plan: "The Design Team is unaware of what specific objectives would be achieved by delaying a KSK roll". The plan was put on hold because of the data from September 2017. At the moment it is unknown if/when there will be a KSK roll. Is not doing a KSK roll by 2020 [1] a viable option? Regards, S. Moonesamy 1. The date is based on a statement in the DPS.

12 19