This came in on the DNS Ops list. Seems like both an interesting potential problem, but also something that is rather pertinent to our work on this WP. Will be interesting to see how this issue evolves!
Begin forwarded message:
Hey Folks,
Has anyone else seen an increase in DNS over TCP traffic in their environment? We have been seeing a steady increase since late last year and have believe we have narrowed down a major cause. After reaching out to the Chromium folks and Cricket Liu reaching out to the Microsoft folks it seems that there has been a recent behavior change that is incompatible with each other, which is causing DNS over TCP to be preferred over UDP.
Based on my discussion with the Chromium team, It appears that for about 3 years Chrome has a bit of internal logic around falling back to TCP when there is a detection of reduced UDP port entropy being handed out by the OS. When the Chrome stack falls back to TCP, according to the Chromium folks, it will continue to use TCP until Chrome is restarted or there is a network change (port flap, IP address change, etc). The code that tracks the low entropy can be found here net::DnsUdpTracker <https://urldefense.com/v3/__https:/nam06.safelinks.protection.outlook.com/?u...>.
The Chromium folks confirmed that they are seeing an increase of TCP traffic from Windows client only. Crickey Liu reach out to the Microsoft folks and dfound that starting with Windows 11, the OS began to use socket caching due to exhaustion occurring with UDP ports. Meaning that DNS UDP port is cached when communicating with the same server and any DNS client will continue to get the same UDP src-port when connecting to that DNS server. Now starting in Chrome 105, there was a change made by the Chromium folks to leverage the internat Chrome DNS stack to to run more Windows DNS queries through the Chrome stack instead of delegating the resolutions to the OS. Due to the low UDP port entropy logic discussed above in combination to the socket caching introduced Windows 11, we are seeing DNS clients preferring TCP over UDP for what seemed like to discernable reason until these discussions with the Chromium and Microsoft folks.
From our perspective this is and will cause a lot of issues for DNS providers as more and more Chrome + Windows clients begin to prefer TCP over UDP for DNS. And believe this has the potential to quickly become a rather large issue for DNS providers, especially at scale.
Is anyone here seeing a seemingly unexplained increase in DNS over TCP traffic and if it is causing any issues within their network?
For reference, Google Chrome version 105 was released on August 30th, 2022 and Windows 11 was released on October 5th, 2021. Only with the combination of the two (post August 30th, 2022) would the issue be seen.
Thanks,
Adam Casella | Solutions Architect Infoblox | infoblox.com <http://infoblox.com/>