Make Connections

Network, Get Answers, Find Members in Your Area, and More!

Forums » Search Engine Optimization & Advertising

what is the role of robot.txt in SEO?

    • 40 posts
    June 14, 2013 12:50 AM EDT

    check out great post on Robots.txt role in seo at webconfs:

    http://www.webconfs.com/what-is-robots-txt-article-12.php

    ---
    Free Website Builder : Create a Free Website for Small Business | WebStartToday.com

    • 36 posts
    July 16, 2013 12:31 AM EDT

    Robot.txt is a text it is used when you want, your website page not indexed or crawled by Google. 

     

     

     

     

    Email.biz

    • 79 posts
  • May 30, 2013 2:01 AM EDT

    Robots.txt file is used to tell the search engines which pages to be indexed by and which pages not to index. Robots.txt is file is used to block the admin pages in the site

    • 5 posts
    June 16, 2013 3:53 AM EDT

    This file is basically used for disallow unnecessary pages.

    ---
    SEO Consulting Services

    • 49 posts
    June 14, 2013 1:18 AM EDT

     

    Robot.txt is a file that tells search robots, which pages you would like to visit and which is not. In simple words it intimates Google not to crawl that particular page.

    • 6 posts
    May 19, 2013 7:18 PM EDT

    That no-indexed webpages do not find yourself within the catalog however banned webpages do, and therefore the latter will appear within the google look for (albeit with less details since the robots cannot see the site content). it's by no indicates that necessary for google however generally google modify what they're requested to not do.

    ---
    seo consulting company

    • 11 posts
    June 14, 2013 9:40 PM EDT

    robots.txt file should be uploaded to your website rood directory and a valid sitemap.xml reference should be detected from it. This file is useful to search engine to know what pages are allowed to be crawled and what are not. Though it looks very small file it is very valuable for SEO.

    • 51 posts
    June 3, 2013 11:41 PM EDT

    Robots.txt is frequent name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site. The robots.txt file is used to have the funds for directions about the Web site to Web robots and spiders.

    ---
    SEO Services | Web Development Services

    • 1 posts
    May 13, 2013 8:00 PM EDT

    Robots.txt contains 2 important information for search engines(Google etc.)

    1) Which pages/areas of site to crawl and index as well as which all pages it should not index and download in their cache, so it can be used for SEO by disallowing indexing of duplicate or dynamic generated urls(with parameters etc), private areas of site which you don't want to appear in search result pages.

    2) Location of sitemap: Robots.txt contains a line that informs search engine about location of sitemap xml or gz files, it will be helpful if site is large and multiple sitemaps are getting used.

    ---
    Try out world's Most Powerful Self managed SEO tool for FREE (Limited offer)

    • 2 posts
    June 4, 2013 9:36 AM EDT

    Every day , some times search engine crawler comes to your site, it will look for a special file on your site. This file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored.

    ---
    air fittings duct flex

    • 30 posts
    May 28, 2013 10:40 PM EDT

    Robot.txt is an on-page SEO technique and it is basically used to allow for the web robots also known as the web wanderers, crawlers or spiders. It is a program that traverses the website automatically and this helps the popular search engine like Google to index the website and its content.

    ---
    Web Development Services | SEO Services

    • 19 posts
    May 28, 2013 11:57 PM EDT

    In simple words, it intimates google not to crawl that particular page.

    ---
    CRM software | sales management software

    • 6 posts
    March 22, 2013 5:18 PM EDT

    Robot.txt contains the instruction for the search engine bots. It has code which instruct search engine bots or crawler about the crawling or indexing website page.It tells bots whether which urls of website it follow or which one not.

    ---
    Virtual Machine Monitoring | Network Monitoring Tools

    • 6 posts
    July 15, 2013 9:00 PM EDT

    Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit.

     

     

     

     

     

    ---
    [url=http://www.bigrock.in/]http://www.bigrock.in/[/url]

    • 16 posts
    May 24, 2013 10:29 PM EDT

     

    Robots.txt is a text file you set on your site to tell search robots which webpages you would like them not to visit. It  helps us to avoid the duplicate links in a website. We clearly mention the links which we want to follow or no follow.

    ---
    Quick and easy ecommerce website optimization from XtremeUX

    • 16 posts
    June 4, 2013 7:24 PM EDT

    Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.

    ---
    Quick and easy ecommerce website optimization from XtremeUX

    • 139 posts
    March 22, 2013 6:22 AM EDT

    Hello Guys,

    Please help me to know more about robot.txt, please tell me that what is that and how its important for best seo result. Thanks for any reply.

     

    ---
    Machine Cargo Moving | Commercial and Business Removals

    • 11 posts
    June 17, 2013 2:21 AM EDT

    Robots.txt is a file through which you can guide search engines to crawl or not to crawl certain sections of your website.

     

    Google specifically follows instructions given in this robots.txt file

    • 37 posts
    May 15, 2013 7:36 PM EDT

    Hi,

    the simplest robots.txt file uses two rules:

    • User-agent: the robot the following rule applies to
    • Disallow: the URL you want to block

    These two lines are considered a single entry in the file. You can include as many entries as you want. You can include multiple Disallow lines and multiple user-agents in one entry.

    ---
    Wordpress Development Company | Wordpress Development Service

    • 10 posts
    May 12, 2013 8:07 PM EDT

    The main use of robot.txt is for the page which you don't want to show to user like admin panel or any 404 page of your website.

     

    ---
    Professional Web Development Company

    • 9 posts
    May 16, 2013 12:43 AM EDT

    The files or directories you do not want to be indexed by search engines, you can use a robots.txt file. It is to tell where the robots should not go. These files are simple text files that are placed on the web server. They must be placed on the root folder like this www.site.com/robots.txt

    And this is how you write:

    User agent: *
    Disallow: /photos

    • 14 posts
    May 15, 2013 12:59 AM EDT

    Robots.txt permits you to specify that pages shouldn't be crawled. Pages that do not get crawled will still rank for keywords and show up in search results. Robots.txt has been with U.S. for over fourteen years, however what percentage people knew that additionally to the require directive there is a no index directive that Google bot obeys? That no-indexed pages do not find yourself within the index however disallowed pages do, and therefore the latter will show up within the search results (albeit with less info since the spiders cannot see the page content). it's by no suggests that necessary for search engines however typically search engines adjust what they're asked to not do.

    ---
    Usda Home Loan | First Time Home Buyer Programs | Usda Rural Loan

    • 3 posts
    June 14, 2013 7:36 PM EDT

    Robot.txt comprises the instruction for the search-engine bots. It has code which instruct search engine bots or crawler about the crawling or indexing site page. It tells bots whether which urls of site it follow or which one not.

    ---
    http://www.pctlc.com/

    • 49 posts
    May 28, 2013 11:25 PM EDT

    Robot.txt is a file that tell search robots which pages you would like to visit or not.