Find us elsewhere
Join Now Member Login

Is there any FREE guide or lesson how to write a robot?

 
New Topic
Post Reply
Follow Topic
Page of 2 Next »
  • Author
  • Message
 
wanhart

posts: 5

Aug 30, 2007 11:22 PM ET    Quote  Report Abuse
Points: 0   Vote
From reading "Will A Sitemap Get Google To Crawl My Site Faster?" in http://www.startupnation.com/forums/6814/1/1. I am adding a site map to my site as soon as I can.

Right now, I wanted to optimize my website ranking and indexing in Live.com (being no.2 website worldwide). At this point I am really concern that my traffic from Live.com is still 0 (zero).

I was reading from Live.com web ranking and indexing guidelines and Live.com suggested to develop a robot.txt. I`ve been to http://www.robotstxt.org/wc/robots.html and it did not provide help in how to write a robot. Many of their books recommendation were written back in the 90`s.

I was wondering if anybody know a website that can give us a FREE complete guidline on how to write a robot.

Irwan.






-------------------------

www.WanHart.com
hostclick

posts: 129

Aug 31, 2007 11:32 AM ET    Quote  Report Abuse
Points: 0   Vote
Generally the robots.txt file is used to limit what content is delivered to a spider/spiders or which spiders can crawl your site.  For example you can use the robots file to tell the internet archive you don`t want them archiving your site.

http://www.archive.org/about/exclude.php

I`ve never heard of it being used in any way to increase rank.
CampSteve

posts: 1216

Aug 31, 2007 1:06 PM ET    Quote  Report Abuse
Points: 0   Vote
Dear Robot,

I am writing you to...


Oh wait, not that kind of writing robots!  Sorry, I couldn`t resist.  :)  Okay people, back to the serious answers to Irwan`s question.
SearchGuy

posts: 6

Aug 31, 2007 2:53 PM ET    Quote  Report Abuse
Points: 0   Vote
MSN/Live.com has been known to over index website pages which may be a reason why they suggest making a robots file. As hostclick stated, it tell bots what to crawl on your site. I`ve used the robots file to purge some files out of the index however not to increase rankings, not sure how that would work. In any case I quickly checked your indexed pages in Live and your site looks fine. In Yahoo however you only have the homepage indexed so I would consider resubmitting.

- Dan


-------------------------

SearchCampaigner - Do-It-Yourself Search Marketing Solutions for small businesses.
malloc

posts: 39

Aug 31, 2007 3:47 PM ET    Quote  Report Abuse
Points: 0   Vote

Irwan,

Please check out http://www.robotstxt.org/wc/norobots.html. This is the standard  for writing and consuming robots.txt. This page also provides several good examples.

The robot.txt file is basically an instruction set that informs web-crawlers what they should / shouldn`t crawl in your site.

There are some sites that provide free, online wizards for this tasks. This is a good one: http://www.mcanerin.com/EN/search-engine/robots-txt.asp

Enjoy,

David

wanhart

posts: 5

Aug 31, 2007 10:20 PM ET    Quote  Report Abuse
Points: 0   Vote
Ok, thanks. I don`t need a robot after all.

But, wait, I may need robot to tell search engine not to crawl to my customer database?


-------------------------

www.WanHart.com
Webline

posts: 687

Aug 31, 2007 10:42 PM ET    Quote  Report Abuse
Points: 0   Vote
Bots crawl directories; I don`t believe theres anyway they can get into a database.

-------------------------

M Hall
Website Critique Community
International Society of Curmudgeons


JoeJustin

posts: 85

Aug 31, 2007 10:46 PM ET    Quote  Report Abuse
Points: 0   Vote
wanahrt,

You do need a robots.txt!  Most folks don`t even know about this file.  The reason why you need this is because if you have an HTML static website you normally want your index page to be the page of entry.  This is the page everyone wants to have the highest Google page rank. 

So here`s why you need a robots.txt file.  Every link you have on your index page, probably every other page on your site will take away from your page rank.  So  let`s say that you have a web site with 7 differnt pages other than your index page.  That means that you will have 7 links off of your index page thus taking away page rank from your site.  What you want to do is to figure out what other paged do you want to have on google besides your home page or index page. 

We can probably say most of the time you don;t really care if your about us page, your contact page or FAQs page shows up in google.  So your robots.txt file will tell google not to spider those pages. 

Using our example we have now shaved off 3 out of the seven pages that hurt your page rank.  contact us, about us and FAQs.  Fi you dig a little deeer you probably can take one or two more off of your list as well.  I hope this helps.  You definitly need a robots.txt file!!!!


-------------------------

Arsenal Marketing
WEB 2.0 Internet Marketing for Business
http://www.arsenalmarketing.com
Joe@arsenalmarketing.com

Reach a larger audience!
Start blogging today!
Using WEB 2.0 strategies & techniques!
nhgnikole

posts: 2660

Sep 01, 2007 1:27 AM ET    Quote  Report Abuse
Points: 0   Vote
OK here`s the top secret answer.

Live search, MSN, etc usually just find stuff. If you are on Google and people are linking to you, Live search will find you.

That being said, here`s the top secret submission page for Live search which is a giant pain to find if you went looking for it on their site.

You have 12 pages indexed by Google. Do a search for site:WanHart.com on Google to see that. What could use some work is the titles and descriptions that are showing up on Google, if search engine traffic is something you are concerned about as part of your overall marketing strategy.
Fred333

posts: 51

Sep 24, 2007 11:01 AM ET    Quote  Report Abuse
Points: 0   Vote
Thanks for the Live link. I was looking for that.

-------------------------

china business card / Wall Street Journal Reader
Page of 2 Next »
Post Reply
 
.
Advertisement

Keep the Community Clean!

  • StartupNation forums should be used as a platform to learn, educate others, share stories, tips & tricks and to provide constructive feedback.
  • Please do not use the Forums for advertising & blatant self-promotion.
  • Please be respectful to other members and refrain from personal attacks and vulgar language.
  • StartupNation reserves the right to delete any message, reply, and/or member who violates our terms of use.
Read full terms of use
Advertisement
Advertisement
Advertisement
Advertisement