What is the robots.txt file?
User-agent: *Disallow: /cgi-bin/Sitemap: http://www.mydomain.com/sitemap.xml.gz
Do I need one?
So what’s all the fuss about?
Do you have a robots.txt file and what’s inside it?
Got anything to hide?
The dangers of a robots.txt file
- If your website is static with no customer information – don’t use one
- Check that you are not disallowing the root folder “/”
- Make sure you disallow any folders that may contain private and sensitive data
- Disallow any folders containing executable web programs
- If your website has a sitemap already generated, add this to the file to help indexing.
- Don’t use comments in the file.