JavaScript Editor Best javascript editor debugger     Freeware editor 
↑


Main Page

202
are welcome and which need to move on to that less-exclusive joint down the street.
Robots.txt
can admit or reject robots on a sitewide, directory-by-directory, or page-by-
page basis.
SEO folks often feel a special affection for the
robots.txt
file because it provides
a rare opportunity to communicate with a search engine robot. However, its capabili-
ties are really very limited.
Robots.txt
files exist only to
exclude
indexing. Just as a
bouncer can keep people out but can’t force anyone to come in, the
robots.txt
file can’t
do anything to entice a robot to spend
more
time or visit
more
pages on your site.
Also, compliance with your
robots.txt
file is voluntary, not mandatory. The major
search engines will generally try to follow your instructions, but other, less-reputable
types might not. This is why you should not rely on your
robots.txt
file to prevent spi-
dering of sensitive, private, or inappropriate materials.
Do You Need a
Robots.txt
File?
You may not need a
robots.txt
file. Without one, all robots will have free access to non-
password-protected pages on your site. To decide if you need a
robots.txt
file for your
website, ask yourself these questions:
• Are there any pages or directories on my site that I do not want listed on the
search engines, such as an intranet or internal phone list?
• Are there any specific search engines that I do not want to display my site?
• Do I know of any dynamic pages or programming features that might cause
problems for spiders, like getting caught in a loop (infinitely bouncing between
two pages)?
• Does my website contain pages with duplicate content? (These should not be
indexed or you may be penalized.)
• Are there directories on the site that contain programming scripts only, not
viewable pages?
If the answers to these questions are no, then you do not need a
robots.txt
file.
You’ve got the rest of the day off! If you have any yes answers, you’ll prepare your
robots.txt
file today.
Create Your
Robots.txt
File
Robots.txt
files are very simple text files. To find a sample, go to
www.yourseoplan.com/
robots.txt
and view ours, or go to just about any other site and look for the
robots.txt
file in the
root directory
.
Now:
Determine whether you need a
robots.txt
file in your website.
c08. 8:08 202


JavaScript Editor Best javascript editor debugger     Freeware editor 
→