Forums: Discussions for entrepreneurs and small business.

Second that.  If you want to look at the robots.txt file of a heavily used site check out Wikipedia`s robot.txt file.

As for the Google/Yahoo Sitemap Generator, try GSiteCrawler this sucker will crawl a given website and create sitemaps for Google and Yahoo.




-------------------------

Aaron Wood
CEO
San Diego Dream Weavers
http://www.sddreamweavers.com
awood@sddreamweavers.com

New and improved! Now with blogging goodness!
http://www.sddreamweavers.com/san-diego-seo-marketing-blog/

ROBOTS is used to block search engines from indexing pages. But many web authors use it to tell search engines to index a page. Here is an example:
<META NAME="robots" CONTENT="ALL">

This tag is a waste of time. If a search engine finds your page and wants to index it, and hasn`t been blocked from doing so, it will. And if it doesn`t want to index a page, it won`t. Telling the search engine to do so doesn`t make a difference.

Here is a special Google meta tag that you can use a couple of ways. Here`s one example:
<META NAME="googlebot" CONTENT="nosnippet">

This meta tag tells Google not to use the description snippet, the piece of information it grabs from within a Web page to use as the description; instead it will use the DESCRIPTION meta tag. Here is another example
<META NAME="googlebot" CONTENT ="noarchive">

Using the  ROBOTS meta tag or the robots.txt file, you can tell the search engines to stay away . The meta tag looks like this:

<META NAME="robots" CONTENT="noindex, nofollow">



-------------------------

AbdelRahman
http://trackthatad.com/?i=137822/f-pif4pabdou-startupnation