Want to scrape Facebook’s site for content? You may want to reconsider how you do so as Facebook has updated their Robots.txt file to be a bit more restrictive. If you aren’t aware of what a Robots.txt file is, you can read more here. Ultimately the Robots.txt file simply restricted certain pages from being indexed by anybody. Now Facebook has become more explicit within the file, limiting indexing to Baidu, Google, MSN, Naver, Slurp, Yandex, and other search engines. The company has also linked to the “Automated Data Collection Terms” page.
Read more
