Svelte CZDS zone files?
It occurs to me that most CZDS users are interested primarily in the collection of "ordinary" records present in the gTLD zone files, and have no intention to e.g. operate secondary servers based on the complete zone data. In particular, the RRSIG and NSEC/NSEC3 records which can take up a significant (perhaps dominant) fraction of the zone file size, may be of little interest to most users. I'd like to propose that ICANN make available a version of each gTLDs zone file that elides the *derived* RRSIG, NSEC and NSEC3 records. Doing this can save significant tranmission costs on the ICANN size, and storage/processing costs on the CZDS user side. If some users want the zone files, RRSIGs and all, those can continue to be available. [ Please DO NOT elide "DS" records, these are not "derived" data, signal which delations are signed, and typically do not take up nearly as much space as RRSIGs. ] -- Viktor.
diffs/deltas might also be a nice thing as well to slim up the bandwidth / storage if we're operating santa's list amendment On Mon, Sep 12, 2022 at 1:00 PM Viktor Dukhovni via gtld-tech < gtld-tech@icann.org> wrote:
It occurs to me that most CZDS users are interested primarily in the collection of "ordinary" records present in the gTLD zone files, and have no intention to e.g. operate secondary servers based on the complete zone data.
In particular, the RRSIG and NSEC/NSEC3 records which can take up a significant (perhaps dominant) fraction of the zone file size, may be of little interest to most users.
I'd like to propose that ICANN make available a version of each gTLDs zone file that elides the *derived* RRSIG, NSEC and NSEC3 records. Doing this can save significant tranmission costs on the ICANN size, and storage/processing costs on the CZDS user side.
If some users want the zone files, RRSIGs and all, those can continue to be available.
[ Please DO NOT elide "DS" records, these are not "derived" data, signal which delations are signed, and typically do not take up nearly as much space as RRSIGs. ]
-- Viktor. _______________________________________________ gtld-tech mailing list gtld-tech@icann.org https://mm.icann.org/mailman/listinfo/gtld-tech
________________________________________________By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.
Thank you Viktor and Jothan, We are currently working on other functional aspects of the system, but we will take note of these suggestions for the team to consider implementing into CZDS in the future. Best regards, --Eduardo A. ICANN From: gtld-tech <gtld-tech-bounces@icann.org> on behalf of Jothan Frakes via gtld-tech <gtld-tech@icann.org> Reply-To: Jothan Frakes <jothan@gmail.com> Date: Monday, September 12, 2022 at 1:48 PM To: "gtld-tech@icann.org" <gtld-tech@icann.org>, Viktor Dukhovni <ietf-dane@dukhovni.org> Subject: Re: [gtld-tech] Svelte CZDS zone files? diffs/deltas might also be a nice thing as well to slim up the bandwidth / storage if we're operating santa's list amendment On Mon, Sep 12, 2022 at 1:00 PM Viktor Dukhovni via gtld-tech <gtld-tech@icann.org> wrote: It occurs to me that most CZDS users are interested primarily in the collection of "ordinary" records present in the gTLD zone files, and have no intention to e.g. operate secondary servers based on the complete zone data. In particular, the RRSIG and NSEC/NSEC3 records which can take up a significant (perhaps dominant) fraction of the zone file size, may be of little interest to most users. I'd like to propose that ICANN make available a version of each gTLDs zone file that elides the *derived* RRSIG, NSEC and NSEC3 records. Doing this can save significant tranmission costs on the ICANN size, and storage/processing costs on the CZDS user side. If some users want the zone files, RRSIGs and all, those can continue to be available. [ Please DO NOT elide "DS" records, these are not "derived" data, signal which delations are signed, and typically do not take up nearly as much space as RRSIGs. ] -- Viktor. _______________________________________________ gtld-tech mailing list gtld-tech@icann.org https://mm.icann.org/mailman/listinfo/gtld-tech ________________________________________________By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.
On Tue, Sep 13, 2022 at 03:09:26AM +0000, Eduardo Alvarez wrote:
We are currently working on other functional aspects of the system, but we will take note of these suggestions for the team to consider implementing into CZDS in the future.
I should perhaps mention that one reason I raised my suggestion now, is that there is distinct possibility that the number of signed .COM zones might rise substantially within the year if not sooner, and the already bulky .COM zome file might become considerably larger. So it may be prudent for ICANN to look for ways to slim it down, and dropping the "derived" DNSSEC records (RRSIG and NSEC3 in this case) would considerably reduce the .COM zone footprint. -- Viktor.
According to Viktor Dukhovni via gtld-tech <ietf-dane@dukhovni.org>:
On Tue, Sep 13, 2022 at 03:09:26AM +0000, Eduardo Alvarez wrote:
We are currently working on other functional aspects of the system, but we will take note of these suggestions for the team to consider implementing into CZDS in the future.
So it may be prudent for ICANN to look for ways to slim it down, and dropping the "derived" DNSSEC records (RRSIG and NSEC3 in this case) would considerably reduce the .COM zone footprint.
I agree that it would make sense to strip the RRSIG and NSEC/NSEC3 records out of the distributed zone files. Before the zone files are distributed, there is already an editing process to remove cruft in the AXFR zone files. (For a while they had comments telling us where the hidden masters were.) It should not be hard to adjust the de-crufter to remove RRSIG and NSEC and NSEC3 as well. R's, John
On 12/09/2022 21:47, Jothan Frakes via gtld-tech wrote:
diffs/deltas might also be a nice thing as well to slim up the bandwidth / storage if we're operating santa's list amendment
Think that deltas/diffs were a suggestion on the original CZDS working group. It is a very efficient method but would add to the time taken to produce the files. Some of the new gTLD zones have very little movenment in a month.
On Mon, Sep 12, 2022 at 1:00 PM Viktor Dukhovni via gtld-tech <gtld-tech@icann.org <mailto:gtld-tech@icann.org>> wrote:
It occurs to me that most CZDS users are interested primarily in the collection of "ordinary" records present in the gTLD zone files, and have no intention to e.g. operate secondary servers based on the complete zone data.
In particular, the RRSIG and NSEC/NSEC3 records which can take up a significant (perhaps dominant) fraction of the zone file size, may be of little interest to most users.
I'd like to propose that ICANN make available a version of each gTLDs zone file that elides the *derived* RRSIG, NSEC and NSEC3 records. Doing this can save significant tranmission costs on the ICANN size, and storage/processing costs on the CZDS user side.
If some users want the zone files, RRSIGs and all, those can continue to be available.
[ Please DO NOT elide "DS" records, these are not "derived" data, signal which delations are signed, and typically do not take up nearly as much space as RRSIGs. ]
It would make the files a lot easier to process and would reduce the bandwidth requirements for the CZDS. Currently, the daily download of zones is around 7GB. Deltas would considerably reduce that download. Regards...jmcc -- ********************************************************** John McCormac * e-mail: jmcc@hosterstats.com MC2 * web: http://www.hosterstats.com/ 22 Viewmount * Domain Registrations Statistics Waterford * Domnomics - the business of domain names Ireland * https://amzn.to/2OPtEIO IE * Skype: hosterstats.com ********************************************************** -- This email has been checked for viruses by Avast antivirus software. www.avast.com
It appears that John McCormac via gtld-tech <jmcc@hosterstats.com> said:
On 12/09/2022 21:47, Jothan Frakes via gtld-tech wrote:
diffs/deltas might also be a nice thing as well to slim up the bandwidth / storage if we're operating santa's list amendment
Think that deltas/diffs were a suggestion on the original CZDS working group. It is a very efficient method but would add to the time taken to produce the files. Some of the new gTLD zones have very little movenment in a month. ...
If people want to go that route, the reasonable approach is to make the files available via rsync. There are too many ways to crreate, publish, and apply diffs already. R's, John
On Sep 13, 2022, at 4:41 PM, John Levine via gtld-tech <gtld-tech@icann.org> wrote:
Think that deltas/diffs were a suggestion on the original CZDS working group. It is a very efficient method but would add to the time taken to produce the files. Some of the new gTLD zones have very little movenment in a month. ...
If people want to go that route, the reasonable approach is to make the files available via rsync. There are too many ways to crreate, publish, and apply diffs already.
Or IXFR. Regards, -drc
If people want to go that route, the reasonable approach is to make the files available via rsync. There are too many ways to crreate, publish, and apply diffs already.
Or IXFR.
CZDS distributes zones as compressed master files. Changing it to work by AXFR/IXFR would be quite a challenge. Rsync would be hard enough to use the existing credentials. Regards, John Levine, johnl@taugh.com, Taughannock Networks, Trumansburg NY Please consider the environment before reading this e-mail. https://jl.ly
On Tue, Sep 13, 2022 at 08:43:13PM -0400, John R Levine via gtld-tech wrote:
CZDS distributes zones as compressed master files. Changing it to work by AXFR/IXFR would be quite a challenge. Rsync would be hard enough to use the existing credentials.
I would not expect rsync to be effective at retrieving deltas of compressed files with randomly scattered changes. Sliding window checksums are unlikely to find many common blocks. Perhaps you're thinking of "BATCH MODE" deltas recorded on the uncompressed before/after images of zone file snapshots? The deltas might be compressible. Again if the density of changes is high enough, rsync may not be the most effective incremental update format. Of course now the client would have to identify the right set of deltas to update from its current state to the desired new state. And, with signed zone files, automated updates of RRSIGs increase the local density of changes, so one might want to elide the RRSIGs in any case. The simplest actionable change would be to drop the RRSIG and NSEC(3) records. -- Viktor.
Using the DNZ protocol does seem traditional here. If ICANN needs help doing DNS, there are a variety of experts around.
-----Original Message----- From: gtld-tech [mailto:gtld-tech-bounces@icann.org] On Behalf Of John R Levine via gtld-tech Sent: Tuesday, September 13, 2022 17:43 To: David Conrad Cc: gtld-tech@icann.org Subject: Re: [gtld-tech] Svelte CZDS zone files?
If people want to go that route, the reasonable approach is to make the files available via rsync. There are too many ways to crreate, publish, and apply diffs already.
Or IXFR.
CZDS distributes zones as compressed master files. Changing it to work by AXFR/IXFR would be quite a challenge. Rsync would be hard enough to use the existing credentials.
Regards, John Levine, johnl@taugh.com, Taughannock Networks, Trumansburg NY Please consider the environment before reading this e-mail. https://jl.ly _______________________________________________ gtld-tech mailing list gtld-tech@icann.org https://mm.icann.org/mailman/listinfo/gtld-tech
________________________________________________By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.
Using the DNS protocol does seem traditional here.
Contracted TLDs provided password-protected FTP downloads of compressed master zone files starting at least two decades ago. CZDS is a putatively better replacement for FTP that avoids every TLD having to manage credentials for every client. A few of the TLDs added in the first rounds still use FTP, including .aero and .post. If you want more background, just ask. R's, John
-----Original Message----- From: gtld-tech [mailto:gtld-tech-bounces@icann.org] On Behalf Of John R Levine via gtld-tech Sent: Tuesday, September 13, 2022 17:43 To: David Conrad Cc: gtld-tech@icann.org Subject: Re: [gtld-tech] Svelte CZDS zone files?
If people want to go that route, the reasonable approach is to make the files available via rsync. There are too many ways to crreate, publish, and apply diffs already.
Or IXFR.
CZDS distributes zones as compressed master files. Changing it to work by AXFR/IXFR would be quite a challenge. Rsync would be hard enough to use the existing credentials.
Regards, John Levine, johnl@taugh.com, Taughannock Networks, Trumansburg NY Please consider the environment before reading this e-mail. https://jl.ly _______________________________________________ gtld-tech mailing list gtld-tech@icann.org https://mm.icann.org/mailman/listinfo/gtld-tech
________________________________________________By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.
Regards, John Levine, johnl@taugh.com, Taughannock Networks, Trumansburg NY Please consider the environment before reading this e-mail. https://jl.ly
On Sep 13, 2022, at 7:20 PM, John R Levine <johnl@taugh.com> wrote:
Using the DNS protocol does seem traditional here. Contracted TLDs provided password-protected FTP downloads of compressed master zone files starting at least two decades ago.
Sure, because that’s what Verisign (Netsol) offered, not because it was a particularly good idea or well thought out.
CZDS is a putatively better replacement for FTP that avoids every TLD having to manage credentials for every client.
CZDS as a credential management system would appear to be independent of how the data access authorized by those credentials is performed.
A few of the TLDs added in the first rounds still use FTP, including .aero and .post.
Yes, a protocol not even the IETF uses any more and which support in browsers is pretty much gone.
If you want more background, just ask.
If it’s relevant, perhaps. Ah well, enough windmill tilting: having some familiarity with these things and no particular dog in this fight anymore, I’ll just go make popcorn. Regards, -drc
On Sep 13, 2022, at 5:43 PM, John R Levine <johnl@taugh.com> wrote:
If people want to go that route, the reasonable approach is to make the files available via rsync. There are too many ways to crreate, publish, and apply diffs already. Or IXFR. CZDS distributes zones as compressed master files.
I know. Despite the fact, IIRC, that (with the exception of .COM) they are fetched by ICANN via zone transfer.
Changing it to work by AXFR/IXFR would be quite a challenge.
On ICANN’s side, what more would it require standing up a name server and sharing TSIG keys? If DNS UPDATE were also implemented, it would address the timeliness issue (if the registries were willing to play along). Of course, CZDS users would likely need to change their code. However, this wouldn’t have to be either/or — both could be done with the benefit of using IXFR being only getting the diffs (and, potentially better timeliness). One can dream… Regards, -drc
Changing it to work by AXFR/IXFR would be quite a challenge.
On ICANN’s side, what more would it require standing up a name server and sharing TSIG keys? If DNS UPDATE were also implemented, it would address the timeliness issue (if the registries were willing to play along).
Due to the three month expiry, I doubt that any two clients have access to the same set of zones, which would make ACL management pretty exciting, particularly since I believe the credentials are stored in some SSO thing from Okta. There's over a thousand zones and there's certainly over a thousand users, so we're talking about ACLs with more than a million entries. Also, based on some of the chatter here, I suspect that a many of of the users do not have the expertise to run a secondary DNS server and manage TSIGs. A lot of CZDS users log into a web site and point and click to download files.
Of course, CZDS users would likely need to change their code. However, this wouldn’t have to be either/or — both could be done with the benefit of using IXFR being only getting the diffs (and, potentially better timeliness).
Viktor and I can do whatever we need to, but I don't think that scales. The automatic scripted stuff is somewhat documented for the daily downloads, and not at all for all the other stuff like extensions and renewals. Regards, John Levine, johnl@taugh.com, Taughannock Networks, Trumansburg NY Please consider the environment before reading this e-mail. https://jl.ly
participants (8)
-
David Conrad -
Eduardo Alvarez -
John Levine -
John McCormac -
John R Levine
-
Jothan Frakes -
Paul Mockapetris -
Viktor Dukhovni