I am confused. Where does robots.txt come into play here? I always thought that ...

		zaro on July 31, 2019 \| parent \| context \| favorite \| on: Web Scraping and Crawling Are Perfectly Legal, Rig... I am confused. Where does robots.txt come into play here? I always thought that if url is allowed in robots.txt it's fair game to scrape.