Internet keyword and search engine marketing strategies to improve your search engine ranking and increase web site traffic.

Getting happy clients first page search engine rankings for over 14 years.


How-To use a search engines robots.txt file

The robots.txt file is a must for your web site. The search engine robots / web crawlers / spiders come looking for this file, and when the robot doesn't find it, he's not a happy spider crawling your site. :-)

example of missing robots.txt file
If you don't have it a 404 error will show up in your logs, as a page not found, a situation you can easily avoid.

Inviting the search engine robots into your site

To ALLOW ALL robots complete access simply copy & paste this code into notepad and save as:

User-agent: *

To EXCLUDE ALL robots from the entire server simply paste this code into notepad and save as

User-agent: *
Disallow: /

NOTE: the robots.txt file Must be on the root level, as shown in the sample urls above. Then you can go check it with a validator like one of these.

robots.txt validator

1. robots.txt syntax checker

Jan, I gotta have more, Ok, ok

HTML Author's Guide to the Robots META tag

What to put into the robots.txt file

There is also a tag you can put into your <head>
<meta name="robots" content="index,follow"> Some question whether this tag has any value, but can't hurt you. There is also a noindex,nofollow or variations of the two.

All robots are welcome to include this page in search services.

This page may not be indexed by a search service.

Robots are welcome to follow links from this page to find other pages.

Robots are not to follow links from this page.

use a robots.txt file and get found

The robots exclusion standard or robots.txt protocol is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website. *ref: robots.txt definition found on Wikipedia