Isn't search engine results a product that Google offers? [1] I find the argument quite strange that website owners agreed to Google being able to do anything with that data beyond displaying them in their search results when they wrote that robots.txt maybe ten years ago, but others shall not access those results programatically.

I certainly did not and find using the content google scraped from my website for money or AI (which they also sell on a token basis) more questionable than some third party offering API access to it.

[1] https://docs.cloud.google.com/generative-ai-app-builder/docs...