| Jul. 06 2006 at 9:49 AM |
|
|
Hey Dan - robots.txt is a file that resides on your web server and whose purpose is to direct the search robots that scan your site for their indexes (google, yahoo, etc.). The robots.txt file's primary purpose is to direct search engine robots as to which content they should avoid indexing on your site - so if you have internally-focused pages or directories, test servers, etc., you don't want those pages indexed and showing up in search results for your site.
chuck fuller
|
| Jul. 06 2006 at 9:54 AM |
|
|
By 'internally-focused pages', does that mean my sensitive .php pages?
Also, I didn't create the robots.txt file and it's not in my directory. Do I have
to add pages to this file and how would I do that?
Dan
|
| Jul. 06 2006 at 10:38 AM |
|
|
It really applies to any pages you wouldn't want someone to find through a search - if I go to Google, for example, and type in a "site: www.yourdomain.com" search, I can see all of the pages you have indexed, and if you didn't exclude pages that might have non-public information, they'd be accessible. I just used the term 'internally-focused' to refer to any pages whose audience is you company, business, etc. - and not the outside world.
You may just have a default robots.txt file on your server automatically - you can edit it with a text editor like Notepad - for information on the setup and commands that go in the file, check out this resource.
chuck fuller
|
| Jul. 06 2006 at 1:05 PM |
|
|
That's very iinteresting. Thanks for the link.
Dan
|
| Jul. 15 2006 at 6:25 AM |
|
|
|
while I was researching the graphics I was in futura publications forum pertaining to anything and everything about web buildings and designs, there was several articles on how the arrangement differs from ms. to google to yahoo and their idea of importance to advertisers and searching clientell. It is interesting reading to some one building a web page and trying to get customers to the site. If interested, that site is
www.desktoppublishingforum.com/bb/showthread.php?t=2370& goto=newpost wiht grace and kinship, we share and all is right with the world. Sara West
|
|
|