Article Categories
» Arts & Entertainment
» Automotive
» Business
» Careers & Jobs
» Education & Reference
» Finance
» Food & Drink
» Health & Fitness
» Home & Family
» Internet & Online Businesses
» Miscellaneous
» Self Improvement
» Shopping
» Society & News
» Sports & Recreation
» Technology
» Travel & Leisure
» Writing & Speaking

  Listed Article

  Category: Articles » Business » Marketing & Promotion » Article

How to keep robots out of your web site

By Dr. Roberto A. Bonomi


You know that search engines have been created to help people find information quickly on the Internet, and the search engines acquire much of their information through robots (also known as spiders or crawlers), that look for web pages for them.

The spiders or crawlers robots explore the web looking for and recording all kinds of information. They usually start with URL submitted by users, or from links they find on the web sites, the sitemap files or the top level of a site.

Once the robot accesses the home page then recursively accesses all pages linked from that page. But the robot can also check out all the pages that can find on a particular server.

After the robot finds a web page it works indexing the title, the keywords, the text, etc. But sometimes you might want to prevent search engines from indexing some of your web pages like news postings, and specially marked web pages (in example: affiliate's pages), but whether individual robots comply to these conventions is pure voluntary.


So if you want robots to keep out from some of your web pages, you can ask robots to ignore the web pages that you don't want indexed, and to do that you can place a robots.txt file on the local root server of your web site.

In example if you have a directory called e-books and you want to ask robots to keep out of it, your robots.txt file should read:

User-agent: * Disallow: e-books/

When you don't have enough control over your server to set up a robots.txt file, you can try adding a META tag to the head section of any HTML document.

In example, a tag like the following tells robots not to index and not to follow links on a particular page:

meta name="ROBOTS" content="NOINDEX, NOFOLLOW"

Support for the META tag among robots is not so frequent as the Robots Exclusion Protocol, but most of major web indexes currently support it.


If you want to keep the search engines out of your news postings, you can create an an "X-no-archive" line in of your postings' headers:

X-no-archive: yes

But although common news clients allow you to add an X-no-archive line to the headers of your news postings, some of them don't permit you to do so.

The problem is that most search engines assume that all information they find is public unless marked otherwise.

So be careful because though the robot and archive exclusion standards may help keep your material out of major search engines there are some others that respect no such rules.

If you're highly concerned about the privacy of your e-mail and Usenet postings, you must use some anonymous remailers and PGP. You can read about it here:

Even if you are not particularly concerned about privacy, remember that anything you write will be indexed and archived somewhere for eternity, so use the robots.txt file as much as you need it.

Written by Dr. Roberto A. Bonomi

About the Author
Dr. Roberto Bonomi is a successful e-book writer that shares his home business experience at: If you already have, or are looking for an Internet Home Business, you can't miss the free knowledge that you'll receive at his site, and you can post free your own articles at

Article Source:
If you wish to add the above article to your website or newsletters then please include the "Article Source:" as shown above and make it hyperlinked.

  Some other articles by Dr. Roberto A. Bonomi
Why should you care about return traffic
The secret of a home business web site's success is the traffic to your web site.. And all you have to do is to attract as many new ...

Spam guard your business
Have you started or are you thinking in starting an easy home business? Then be carefull because spam guard may harm your business! ...

Targeted Traffic Is The Secret For Good Sales!
What's a targeted traffic made of? The people that wants to find on Internet, what you are offering on your web site, and visit your web site, are called targeted traffic ...

Are Top Web Sites Good For Everybody?
Top web sites are site lists focused around a common topic, for example clip art, pets, music etc. Sites on these lists are ranked in order of "votes" - ...

The Viral Marketing Secrets Free Ebook
A free viral marketing system's knowledge can Change Your Online Business Forever... From the Computer of Dr. Bonomi Re: The Free Viral Marketing System's Secrets eBook Dear Friend, This a very special ...

Karma versus Meditation
The Karma lives with us, the karma is part of our lives from the very moment of our birth to the moment of our death, because every time that we do something and even every time ...

  Recent Articles
The Affiliate Marketing Network Advantage
by Laurie Raphael

Marketing your business online
by Candy Steele

Public Relations
by Ismael D. Tabije

Thirteen Step Action Plan For Everyone, That Needs More Business Now.
by Paul Douglas

Article Marketing & List Building: How to Promote Your Ezine & Build Your Own Hyper-Responsive List
by Eric Gruber

How To Build An Opt In List And Your Business
by Dencho Denchev

4 color printing in business cards and posters; You cannot have it any other way
by Florie Lyn Masarate

Plumbing marketing approaches that make your business work with a profit
by Ken Wilson

Builders projects in India
by yaken schecher

What You Should Know To Build Your Affiliate Web Site
by Laurie Raphael

Professional Logo Design: The Foundation To A Powerful Brand
by Alfred Anderson

Equipment, cost and communication; What good printers are made of
by Florie Lyn Masarate

Can't connect to database