- dns-stats-users - lists.icann.org

DNS-STATS Compactor 1.2.1-rc2 release
by Sara Dickinson Feb. 21, 2022

Feb. 21, 2022

Hi all, We are pleased to announce the release of DNS-STATS Compactor version 1.2.1-rc2. Updates in 1.2.1-rc2 since 1.2.1-rc1 * The behaviour of `compactor` is changed when an interrupt (SIGINT, SIGTERM) is received e.g. when the service is restarted or a user enters ^C. Previously all of the in progress C-DNS compression was aborted and all of the in progress output files (compressed and uncompressed) were removed. Now the `.raw` uncompressed output file is left on disk so it can be manually recovered (i.e. renamed and compressed). Users will now need to choose how to managed these uncompressed files. Behaviour on SIGHUP (re-reading of config file without affecting any in progress compression) is unchanged. * Additional default logging is added of the specific signal received by `compactor`, when file rotation occurs and when compression is aborted. * New `log-file-handing` option to log low level details of file processing on file rotation and compression. This aids with debugging of file processing problems and measurement of C-DNS file compression times. * Text added in the documentation to clarify time patterns are needed in file output patterns for filename based file rotation to occur (based on the `rotation-period` option). * Fix a bug relating to `compactor` start_time handling introduced in rc1. Updates in 1.2.1-rc1 since 1.2.1-beta2 * Modify the format output by the debug option `--debug-qr` to be more human readable (`--debug-dns` output is left unchanged). * Fix 2 minor bugs where an incorrect start time could be written to the CDNS file, resulting in a negative file duration. * Documentation: correct the dependency list. * Fix build issue when using `configure --disable-pseudo-anonymisation`. Thanks Sean Burford for reporting this. * Add example docker image as a submodule in `contrib/dns-stats-docker`. Thanks to Sean Burford for providing this. Updates in 1.2.1-beta2 since 1.2.1-beta1: * Bump the private version because schema extended with new stats. * Add new stats to C-DNS: sniffer drops, matcher drops and sampling discards, and 3 libpcap stats. The libpcap stats should not be considered reliable and provided for information only. * Fix file durations for files with more than one C-DNS block in them so high traffic rates can be correctly calculated from the statistics. * Constrain size of matcher queue to reduce memory consumption when dropping/sampling. * Add `-D` option (--relaxed-mode). With this option a warning (not an error) is issued for unrecognized command line and config file options. Updates in 1.2.1-beta1 since 1.2.0: * Change behaviour of internal sniffer channel when overloaded to better report and manage dropping/sampling of packets * This fixes a bug where the 'Total Packets processed' statistic did not include any packets dropped during overload * Improve `log-interval` statistics and packet drop logging * EXPERIMENTAL: New sampling mode option (disabled by default) * sampling can be enabled at a specified rate for a specified time when packet drops exceed a specified threshold (see config options) *Please note the change of PPA - Bionic packages are available for this release* For more information, please see the project wiki: https://github.com/dns-stats/compactor/wiki. Packages available from: ppa:dns-stats/compactor-unstable-bionic Source code available at: https://github.com/dns-stats/compactor Documentation: https://github.com/dns-stats/compactor/wiki/user-guide-1.2.1-rc2.pdf Sara.

1 0

accept / ignore-rr-type options,
by Thib D Feb. 10, 2022

Feb. 10, 2022

Hello, I am having trouble understanding what these options are accomplishing and how to use them to only capture outgoing AXFR queries Shouldn't either one of these do the job ? accept-rr-types=AXFR or ignore-rr-type=<rrtype> for each rrtype? After using inspector to display all of the logged queries in the cdns file, I am still able to see every other query of any rrtype. What am I using wrong here ? Regards, Thibaud

2 1

DNS-STATS Compactor 1.2.1-beta2 release
by Sara Dickinson Jan. 19, 2022

Jan. 19, 2022

Hi all, We are pleased to announce the release of DNS-STATS Compactor version 1.2.1-beta2. Updates since 1.2.1-beta1: * Bump the private version because schema extended with new stats. * Add new stats to C-DNS: sniffer drops, matcher drops and sampling discards, and 3 libpcap stats. The libpcap stats should not be considered reliable and provided for information only. * Fix file durations for files with more than one C-DNS block in them so high traffic rates can be correctly calculated from the statistics. * Constrain size of matcher queue to reduce memory consumption when dropping/sampling. * Add `-D` option (--relaxed-mode). With this option a warning (not an error) is issued for unrecognized command line and config file options. *Please note the change of PPA - Bionic packages are available for this release* For more information, please see the project wiki: https://github.com/dns-stats/compactor/wiki. Packages available from: ppa:dns-stats/compactor-unstable-bionic Source code available at: https://github.com/dns-stats/compactor Documentation: https://github.com/dns-stats/compactor/wiki/user-guide-1.2.1-beta2.pdf Sara.

1 2

Running collector on Kubernetes
by Sean Burford Nov. 16, 2021

Nov. 16, 2021

Hi, I'm running NSD and Unbound on Kubernetes, and I wanted to run the DNS Stats Collector alongside them. To this end I created a Docker image for Collector (https://github.com/SeanBurford/dns-stats-docker) I'm sharing it here in case others find it useful. There are README.md files throughout the tree recording my thoughts, but here are the main points: - Dockerfile ( https://github.com/SeanBurford/dns-stats-docker/blob/main/compactor/Dockerf…) contains the directions used to build the binaries and then the container. - bin/start_compactor.sh adds the hostname to the output file template, which is handy when you're running multiple instances on Kubernetes (they all have unique hostnames). - I use a Unix socket to pass dnstap messages from the name servers to the compactors, with one compactor instance per name server instance. - I'm using NFS to mount a shared log directory across the instances, though there are plenty of options for cluster filesystems. - I would have preferred to use Alpine Linux, however various dependencies made Ubuntu (Focal) an easier choice. This is fine, Alpine Linux containers run alongside Ubuntu containers in a single Kubernetes pod (sharing the dnstap Unix socket). Feel free to send pull requests or suggestions. -- Thanks, Sean Burford

1 0

DNS-STATS Compactor 1.2.1-beta1 release
by Sara Dickinson Aug. 26, 2021

Aug. 26, 2021

Hi all, We are pleased to announce the release of DNS-STATS Compactor version 1.2.1-beta1. Updates since 1.2.0: * Change behaviour of internal sniffer channel when overloaded to better report and manage dropping/sampling of packets * This fixes a bug where the 'Total Packets processed' statistic did not include any packets dropped during overload * Improve `log-interval` statistics and packet drop logging * EXPERIMENTAL: New sampling mode option (disabled by default) * sampling can be enabled at a specified rate for a specified time when packet drops exceed a specified threshold (see config options) *Please note the change of PPA - Bionic packages are available for this release* For more information, please see the project wiki: https://github.com/dns-stats/compactor/wiki. Packages available from: ppa:dns-stats/compactor-unstable-bionic Source code available at: https://github.com/dns-stats/compactor Documentation: https://github.com/dns-stats/compactor/wiki/user-guide-1.2.1-beta1.pdf Sara.

1 0

DNS-STATS Compactor 1.2.0 release
by Sara Dickinson May 10, 2021

May 10, 2021

Hi all, We are pleased to announce the release of DNS-STATS Compactor version 1.2.0. Updates since 1.1.0: * Add capture from DNSTAP, either over Unix socket or from file. * Add custom field of collection start time to block and to info file. File duration in info file is now end-start time. * Correct stats reporting processed message count to RFC8618. Retain total packet count as compactor-specific item. * Add KNOWN ISSUE about the observed performance on VMs being much lower than expected, which is under investigation. * Get revised signal handling building on Mac as well as Linux and FreeBSD. * If configured service address does not exist on any interface log a warning not an error. * Minor test and documentation updates. For more information, please see the project wiki: https://github.com/dns-stats/compactor/wiki <https://github.com/dns-stats/compactor/wiki>. Packages available from: ppa:dns-stats/compactor Source code available at: https://github.com/dns-stats/compactor <https://github.com/dns-stats/compactor> Documentation: https://github.com/dns-stats/compactor/wiki/user-guide.pdf <https://github.com/dns-stats/compactor/wiki/user-guide.pdf> Sara Dickinson.

1 0

DNS-STATS Compactor 1.2.0-rc1 release
by Jim Hague April 19, 2021

April 19, 2021

Hi all, We are pleased to announce the release of DNS-STATS Compactor version 1.2.0-rc. This release is the first release candidate for version 1.2.0. Changes from version 1.1.0 are: * Added capture via DNSTAP from file or Unix socket. * One new optional implementation-specific field in the C-DNS block preamble, recording the collection start time. * Correct value recorded in C-DNS as processed message count. In previous versions this was, erroneously, the raw packet count. The raw packet count is still recorded as a new implementation-specific field. Changes from version 1.2.0-beta4 are: * Fix building revised signal handling on Mac. * Minor test and documentation updates. * Correct processed message count. For more information, please see the project wiki: https://github.com/dns-stats/compactor/wiki. Packages for EXPERIMENTAL releases available from: ppa:dns-stats/compactor-unstable Source code available at: https://github.com/dns-stats/compactor Documentation available at: https://github.com/dns-stats/compactor/wiki/user-guide-1.2.0-rc1.pdf https://github.com/dns-stats/compactor/wiki/user-guide-1.2.0-rc1.html -- Jim Hague - jim(a)sinodun.com Never trust a computer you can't lift.

1 0

DNS-STATS Compactor 1.2.0-beta4 release
by Jim Hague April 12, 2021

April 12, 2021

Hi all, We are pleased to announce the release of DNS-STATS Compactor version 1.2.0-beta4. This release is the second public beta for version 1.2.0. Changes from version 1.1.0 are: * Added capture via DNSTAP from file or Unix socket. * One new optional implementation-specific field in the C-DNS block preamble, recording the collection start time. Changes from version 1.2.0-beta3 are: * Revise signal handling for improved reliability with DNSTAP. * Handle malformed packets in DNSTAP input. * Error given when configured server address not found on any current interface downgraded to a warning. For more information, please see the project wiki: https://github.com/dns-stats/compactor/wiki. Packages for EXPERIMENTAL releases available from: ppa:dns-stats/compactor-unstable Source code available at: https://github.com/dns-stats/compactor Documentation available at: https://github.com/dns-stats/compactor/wiki/user-guide-1.2.0-beta4.pdf https://github.com/dns-stats/compactor/wiki/user-guide-1.2.0-beta4.html -- Jim Hague - jim(a)sinodun.com Never trust a computer you can't lift.

1 0

DNS-STATS Compactor 1.2.0-beta3 release
by Jim Hague March 30, 2021

March 30, 2021

Hi all, We are pleased to announce the release of DNS-STATS Compactor version 1.2.0-beta3. This release is the first public beta for version 1.2.0. Changes from version 1.1.0 are: * Added capture via DNSTAP from file or Unix socket. * One new optional implementation-specific field in the C-DNS block preamble, recording the collection start time. For more information, please see the project wiki: https://github.com/dns-stats/compactor/wiki. Packages for EXPERIMENTAL releases available from: ppa:dns-stats/compactor-unstable Source code available at: https://github.com/dns-stats/compactor Documentation available at: https://github.com/dns-stats/compactor/wiki/user-guide-1.2.0-beta3.pdf https://github.com/dns-stats/compactor/wiki/user-guide-1.2.0-beta3.html -- Jim Hague - jim(a)sinodun.com Never trust a computer you can't lift.

1 0

DNS-STATS Visualizer 1.0.0 release
by Jim Hague March 3, 2021

March 3, 2021

Hi all, We are pleased to announce the initial open source release of DNS-STATS Visualizer version 1.0.0. The DNS-STATS Visualizer is a framework for a system which can: * Consume DNS traffic data files recorded in Compacted-DNS (C-DNS, RFC8618) format from nameservers. * Populate a ClickHouse database with per query/response level data (and additionally aggregate data at a chosen time interval) * Produce DSC-like statistics graphs of the recorded traffic in Grafana (see Screenshots). For more information, please see the project wiki: https://github.com/dns-stats/visualizer/wiki and also this recent presentation at OARC 33: https://indico.dns-oarc.net/event/34/contributions/788/ Packages are available from: ppa:dns-stats/visualizer Source code available at: https://github.com/dns-stats/visualizer Documentation available at: https://github.com/dns-stats/visualizer/wiki/Overview_and_Basic_Install.pdf https://github.com/dns-stats/visualizer/wiki/Overview_and_Basic_Install.html -- Jim Hague jim(a)sinodun.com Never trust a computer you can't lift. Sinodun Internet Technologies Ltd. +44 7941 697732

1 0