Forums: Discussions for entrepreneurs and small business.

From reading "Will A Sitemap Get Google To Crawl My Site Faster?" in http://www.startupnation.com/forums/6814/1/1. I am adding a site map to my site as soon as I can.

Right now, I wanted to optimize my website ranking and indexing in Live.com (being no.2 website worldwide). At this point I am really concern that my traffic from Live.com is still 0 (zero).

I was reading from Live.com web ranking and indexing guidelines and Live.com suggested to develop a robot.txt. I`ve been to http://www.robotstxt.org/wc/robots.html and it did not provide help in how to write a robot. Many of their books recommendation were written back in the 90`s.

I was wondering if anybody know a website that can give us a FREE complete guidline on how to write a robot.

Irwan.






-------------------------

www.WanHart.com

Generally the robots.txt file is used to limit what content is delivered to a spider/spiders or which spiders can crawl your site.  For example you can use the robots file to tell the internet archive you don`t want them archiving your site.

http://www.archive.org/about/exclude.php

I`ve never heard of it being used in any way to increase rank.

Dear Robot,

I am writing you to...


Oh wait, not that kind of writing robots!  Sorry, I couldn`t resist.  :)  Okay people, back to the serious answers to Irwan`s question.

MSN/Live.com has been known to over index website pages which may be a reason why they suggest making a robots file. As hostclick stated, it tell bots what to crawl on your site. I`ve used the robots file to purge some files out of the index however not to increase rankings, not sure how that would work. In any case I quickly checked your indexed pages in Live and your site looks fine. In Yahoo however you only have the homepage indexed so I would consider resubmitting.

- Dan


-------------------------

SearchCampaigner - Do-It-Yourself Search Marketing Solutions for small businesses.

Irwan,

Please check out http://www.robotstxt.org/wc/norobots.html. This is the standard  for writing and consuming robots.txt. This page also provides several good examples.

The robot.txt file is basically an instruction set that informs web-crawlers what they should / shouldn`t crawl in your site.

There are some sites that provide free, online wizards for this tasks. This is a good one: http://www.mcanerin.com/EN/search-engine/robots-txt.asp

Enjoy,

David

Ok, thanks. I don`t need a robot after all.

But, wait, I may need robot to tell search engine not to crawl to my customer database?


-------------------------

www.WanHart.com

Bots crawl directories; I don`t believe theres anyway they can get into a database.

-------------------------

M Hall
Website Critique Community
International Society of Curmudgeons

wanahrt,

You do need a robots.txt!  Most folks don`t even know about this file.  The reason why you need this is because if you have an HTML static website you normally want your index page to be the page of entry.  This is the page everyone wants to have the highest Google page rank. 

So here`s why you need a robots.txt file.  Every link you have on your index page, probably every other page on your site will take away from your page rank.  So  let`s say that you have a web site with 7 differnt pages other than your index page.  That means that you will have 7 links off of your index page thus taking away page rank from your site.  What you want to do is to figure out what other paged do you want to have on google besides your home page or index page. 

We can probably say most of the time you don;t really care if your about us page, your contact page or FAQs page shows up in google.  So your robots.txt file will tell google not to spider those pages. 

Using our example we have now shaved off 3 out of the seven pages that hurt your page rank.  contact us, about us and FAQs.  Fi you dig a little deeer you probably can take one or two more off of your list as well.  I hope this helps.  You definitly need a robots.txt file!!!!


-------------------------

Arsenal Marketing
WEB 2.0 Internet Marketing for Business
http://www.arsenalmarketing.com
Joe@arsenalmarketing.com

Reach a larger audience!
Start blogging today!
Using WEB 2.0 strategies & techniques!

OK here`s the top secret answer.

Live search, MSN, etc usually just find stuff. If you are on Google and people are linking to you, Live search will find you.

That being said, here`s the top secret submission page for Live search which is a giant pain to find if you went looking for it on their site.

You have 12 pages indexed by Google. Do a search for site:WanHart.com on Google to see that. What could use some work is the titles and descriptions that are showing up on Google, if search engine traffic is something you are concerned about as part of your overall marketing strategy.

Thanks for the Live link. I was looking for that.

-------------------------

china business card / Wall Street Journal Reader