Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

The TOS that Google follows is published in the robots.txt file. If you don't want Google to scrape your site, then that's all you need. There's no double standard.


I'm sure that's true for your average Wordpress publisher, but the big guys will either slap you with a law suit or take other measures to make you stop crawling their site.

Scraping and crawling is the same thing btw. I absolutely love how the English language has several words for the same thing. Your language very expressive.

Google is a scraper. Your data will end up in their index. You are perfectly OK with Google "stealing" your data.

A new player crawling your site is an offence to you. How dare someone other than Google or Bing put preasure on my site? How dare they steal my data?

TOS is a joke.

I wonder, what was the intention of the founding fathers of the internet, of the internet? Was it not to make data publicly available?




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: