Hi All, The robot.txt file is encoded in UTF-8 format and I am just curious if there has been any study or analysis done on web crawlers behavior on UA . Kind regards, Dessalegn
This is somewhat related to search engines I suppose… I believe there were some that were looked at before. But crawler tools themselves, I am not sure. But yes, a good point of analysis. Edmon From: UA-discuss [mailto:ua-discuss-bounces@icann.org] On Behalf Of Dessalegn Yehuala Sent: Wednesday, January 8, 2020 3:26 PM To: UA-discuss <UA-discuss@icann.org> Subject: [UA-discuss] Web Crawlers behavior on UA Hi All, The robot.txt file is encoded in UTF-8 format and I am just curious if there has been any study or analysis done on web crawlers behavior on UA . Kind regards, Dessalegn
On 8 Jan 2020, at 2:26, Dessalegn Yehuala wrote:
Hi All,
The robot.txt file is encoded in UTF-8 format and I am just curious if there has been any study or analysis done on web crawlers behavior on UA .
good point. also the sitemap are to be considered. Marc.
Kind regards,
Dessalegn
_______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.
participants (3)
-
Dessalegn Yehuala -
Edmon -
Marc Blanchet