Make Connections

Network, Get Answers, Find Members in Your Area, and More!

Forums » Search Engine Optimization & Advertising

what is the role of robot.txt in SEO?

    • 3 posts
    January 28, 2015 3:57 AM EST
    Robot file is use to block some specific URLs of the website. It guide Google robot not to index some specific URls of the website.

    Regards,
    Jerry @ http://www.vinitysoft.com/tool-asset-management-software/
    • 45 posts
    December 8, 2015 11:51 PM EST
    When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored.
    The robots.txt file is a simple text file that must be placed in your root directory, for example:
    http://www.yourwebsite.com/robots.txt