synthetic data: Expectations vs. Reality

https://www.instapaper.com/read/1717250761

This allows Goggle (consumer-agent title GoogleBot) to visit each and every webpage and directory, whilst at the same time banning Check with Jeeves from your internet site totally