I’d like to discuss benefits of advertising user agent automation. I started the topic at webappsec (https://lists.w3.org/Archives/Public/public-webappsec/2017Jan/0004.html), but this mailing list seems to be more suitable for the topic.
The idea is to attach an HTTP request header to navigation requests that are initiated by automation tools, by which I mean headless browsers, web driver driven browsers, etc.
The benefit for the webste operator is to have a choice in responding to such requests differently. For example, do not serve ads, suggest using API scraping rather that loading heavy resources, send through failed CAPTCHA route, etc.
This approach intersects with robots.txt a little, but none of modern UA automation tools honor robots.txt, and implementing the advertising flag seems to be relatively easy.