Jan. 8, 2020
7:35 a.m.
This is somewhat related to search engines I suppose… I believe there were some that were looked at before. But crawler tools themselves, I am not sure. But yes, a good point of analysis. Edmon From: UA-discuss [mailto:ua-discuss-bounces@icann.org] On Behalf Of Dessalegn Yehuala Sent: Wednesday, January 8, 2020 3:26 PM To: UA-discuss <UA-discuss@icann.org> Subject: [UA-discuss] Web Crawlers behavior on UA Hi All, The robot.txt file is encoded in UTF-8 format and I am just curious if there has been any study or analysis done on web crawlers behavior on UA . Kind regards, Dessalegn