Create Attention Grabbing, Search Engine Friendly robots.txt file Fast and Easy!
Disallow and Allow Robot Access
Major search engines (for example, Google, Bing and Yahoo!) allow robots to index files and folders. The Allow directive works just the opposite of Disallow.
You can specify which content will be disallowed and allowed to be crawled.
You can define between which hours of the day or night you want your pages to be crawled
If a robot is crawling your site too frequently and chewing up your bandwidth, use Crawl Delay. This will reduce the load on your server and help your webpages open faster.
You can define how many pages for robots to crawl per second.
You can show robots where your sitemap is located for quick and easy indexing.
User Agents List
A user agent is any device that interprets web documents. User agents include visual browsers (text-only and graphical), non-visual browsers (audio, braille), and search engine robots.
A list ofUser Agents shown to index your files and folders are included in the program easy access. Alternatively, you can enter your own favorite robot.
Now it's easy to instruct robots which of your files and folders to allow or exclude from indexing!
"FTP Access" is included !
"FTP Access" is included for easy drag & drop or copy/paste robots.txt file Upload. Once uploaded to the root directory of your website files, it starts to work immediately!
Whenever the next robot comes to your site, it will first look at your robots.txt file to see which files it is allowed or not allowed to index with a crawl delay if any, and allow any sitemap(s), and then will go further to index all files and folders that were not listed in the robots.txt file.
If you don't have a valid robots.txt file, robots will read and index every file and folder they can get their hands on !
Valid Robots.txt Files
Robots.txt Maker produces a valid coded robots.txt file that is stored with your website files. Its purpose is to give instructions to user agents (also known as "robots", "crawlers" or "spiders") detailing what they should or should not index on your website.
As long as you have some basic knowledge of how to edit and upload your webpage, you'll be glad you have Robots.txt Maker.
Be sure search engine spiders index the files you want with Robots.txt Maker !
Download the FREE demo version NOW!