Jan. 29, 2016
4:49 p.m.
On 29 Jan 2016, at 5:34, Hollenbeck, Scott wrote:
It also begs the question of the need for a BCP describing operational practices for server operators. There are ways for web servers to influence or restrict crawler behavior, but what's appropriate in this context?
It would be good to keep in mind that respecting mechanisms such as robots.txt is entirely voluntary on the crawler’s side. Luis Muñoz Director, Registry Operations ____________________________ http://www.uniregistry.link/ 2161 San Joaquin Hills Road Newport Beach, CA 92660 Office +1 949 706 2300 x 4242 lem@uniregistry.link