Internet keyword and search engine marketing strategies to improve your search engine ranking and increase web site traffic.

Getting happy clients first page search engine rankings for over 14 years.
 

How-To use a search engines robots.txt file

The robots.txt file is a must for your web site. The search engine robots / web crawlers / spiders come looking for this file, and when the robot doesn't find it, he's not a happy spider crawling your site. :-)

example of missing robots.txt file
If you don't have it a 404 error will show up in your logs, as a page not found, a situation you can easily avoid.

Inviting the search engine robots into your site

To ALLOW ALL robots complete access simply copy & paste this code into notepad and save as: http://www.yourdomain.com/robots.txt

User-agent: *
Disallow:

To EXCLUDE ALL robots from the entire server simply paste this code into notepad and save as http://www.yourdomain.com/robots.txt

User-agent: *
Disallow: /

NOTE: the robots.txt file Must be on the root level, as shown in the sample urls above. Then you can go check it with a validator like one of these.

robots.txt validator

1. robots.txt syntax checker

Jan, I gotta have more, Ok, ok

HTML Author's Guide to the Robots META tag

What to put into the robots.txt file

There is also a tag you can put into your <head>
<meta name="robots" content="index,follow"> Some question whether this tag has any value, but can't hurt you. There is also a noindex,nofollow or variations of the two.

INDEX
All robots are welcome to include this page in search services.

NOINDEX
This page may not be indexed by a search service.

FOLLOW
Robots are welcome to follow links from this page to find other pages.

NOFOLLOW
Robots are not to follow links from this page.

use a robots.txt file and get found

The robots exclusion standard or robots.txt protocol is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website. *ref: robots.txt definition found on Wikipedia

copyright © www.Keyphrase-Marketing.com all rights reserved
321 harwood street elyria, ohio 44035 u.s.a. ph: 440.365.8034
located in lorain county, 30 miles west of cleveland, oh
your comments and questions are welcome click help@keyphrase-marketing.com