On 8 Jan 2020, at 2:26, Dessalegn Yehuala wrote:
Hi All,
The robot.txt file is encoded in UTF-8 format and I am just curious if there has been any study or analysis done on web crawlers behavior on UA .
good point. also the sitemap are to be considered. Marc.
Kind regards,
Dessalegn
_______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.