The robot.txt file is encoded in UTF-8 format and I am just curious if there has been any study or analysis done on  web crawlers behavior on UA .

Kind regards,

Dessalegn