On 29 Jan 2016, at 5:34, Hollenbeck, Scott wrote:

It also begs the question of the need for a BCP describing operational practices for server operators. There are ways for web servers to influence or restrict crawler behavior, but what's appropriate in this context?

It would be good to keep in mind that respecting mechanisms such as robots.txt is entirely voluntary on the crawler’s side.