What is a Robots.txt file and how to create?

What is a simple robots.txt file?
Please note: This will allow all robots to crawl and index all files.

This allows all robots to crawl all files.

User-agent: *

What if I don’t want a particular file crawled?
Please note: Disallowing a specific file to be crawled will keep it from being indexed. The file disallowed will not show up in the search engines. But , this is only effective for friendly robots. Robots can choose to ignore your instructions.

This allows all robots to crawl all files except the images file.

User-agent: *
Disallow: /images/

This allows all robots to crawl all files except the images file and the stats file.

User-agent: *
Disallow: /images/
Disallow: /stats/

What if I want to disallow a particular robot?
Sometimes, you may find that you would like to disallow specific robots from crawling your site or limit which files they may have access to.

This denies access to Googlebot-image to any files in your domain

User-agent: Googlebot-Image
Disallow: /

This specifically denies Googlebot-image to your images file

User-agent: Googlebot-Image
Disallow: /images/

For a current data base of robot names and information, visit:

Note: pelase upload the robots.txt to your website root




One Response

  1. Lacie Muhs January 17, 2010