Don't crawl me so fast, bro
This weekend I had a developer contact to see if my crawler for Route285.com respected the crawl-delay directive in robots.txt. It does. I thought this was a cool bit of HTTP working well. My crawler sets it's User Agent header, so the site owner knew who to reach