is a visual editor for Robot Exclusion Files. It allows
the user to quickly and easily create the ROBOTS.TXT files required
to instruct web search engines which parts of a web site are
not to be indexed and made searchable by the general web public.
RoboGen does this by providing the user a way to log onto his
FTP server and then select the documents and directories which
are not to be made searchable.
information on RoboGen and robot exclusion files, see
Introduction to Web Robots, ROBOTS.TXT,
For a detailed
overview of RoboGen's features, see the RoboGen
Feature Comparison Chart.
Robot Exclusion Files by selecting documents and directories.
into FTP servers and even upload robots.txt from RoboGen.
information for multiple servers.
of over 180 know user-agents and 10 major search engines.
to edit the robot database and add additional user-agents.
recognizes common document roots when logging into FTP servers
and suggests that they be set as the document root.
is available in two editions, Standard and Limited. RoboGen
LE is a bare bones version that simply creates new Robot Exclusion
Files (ROBOTS.TXT). It lacks some key abilities such as
opening existing files and ability to upload directly to the
FTP server. RoboGen SE is the more advanced, full featured
version of the software. It can do everything LE can plus
it can do many things LE cannot, including opening existing
files and uploading files directly to the server. A free
trial version of RoboGen SE is available and it only costs $12.95
to purchase a registered copy. Information on how to purchase
the RoboGen software is available in the RoboGen SE package.
For a detailed
overview of the differences between RoboGen LE and SE, see the
RoboGen Feature Comparison Chart.