Introduction to RoboGen

RoboGen is a visual editor for Robot Exclusion Files.  It allows the user to quickly and easily create the ROBOTS.TXT files required to instruct web search engines which parts of a web site are not to be indexed and made searchable by the general web public.  RoboGen does this by providing the user a way to log onto his FTP server and then select the documents and directories which are not to be made searchable.

For more information on RoboGen and robot exclusion files, see  Introduction to Web Robots, ROBOTS.TXT, and RoboGen.

Feature Overview

• Create Robot Exclusion Files by selecting documents and directories.
• Log into FTP servers and even upload robots.txt from RoboGen.
• Manage information for multiple servers.
• Database of over 180 know user-agents and 10 major search engines.
• Ability to edit the robot database and add additional user-agents.
• RoboGen recognizes common document roots when logging into FTP servers and suggests that they be set as the document root.

Pricing

RoboGen is available in two editions, Standard and Limited.  RoboGen LE is a bare bones version that simply creates new Robot Exclusion Files (ROBOTS.TXT).  It lacks some key abilities such as opening existing files and ability to upload directly to the FTP server.  RoboGen SE is the more advanced, full featured version of the software.  It can do everything LE can plus it can do many things LE cannot, including opening existing files and uploading files directly to the server.  A free trial version of RoboGen SE is available and it only costs $12.95 to purchase a registered copy.  Information on how to purchase the RoboGen software is available in the RoboGen SE package.

For a detailed overview of the differences between RoboGen LE and SE, see the RoboGen Feature Comparison Chart.