Google Adds DIY Robots.txt creator to Webmaster Tools

For those of you unfamiliar with Robots.txt files, Wikipedia explains them:

A robots.txt file on a website will function as a request that specified robots ignore specified files or directories in their search. This might be, for example, out of a preference for privacy from search engine results, or the belief that the content of the selected directories might be misleading or irrelevant to the categorization of the site as a whole, or out of a desire that an application only operate on certain data.

Now if you're not overly confident creating your robots.txt files from scratch, you can simply create one using your Google Webmaster Tools.

As the Google release post suggests:

"It's designed to give you an easy and interactive way to build a robots.txt file. It can be as simple as entering the files and directories you don't want crawled by any robots."

The tool allows users to either create broad exclusion rules which apply to all robots and crawlers or specific rules for particular robots. See the screenshots below.

Once you've created the robots.txt file using the tool, all you have to do is upload it to the top level directory (root) of your website and voila - you're all done.

As the release post highlights, not all robots obey the conventions of the Robots.txt protocol, but rest assured most of the major search engines do, which is where most of your website crawling will come from.

Posted by Rene LeMerle at 9:31 AM GMT