All web sites need a good robots.txt in my opinion. For those reading (Ha….what a joke!) who don’t know what that is, a robots.txt file is simply a text file that tells search engines what they are allowed to tell the world about, and what you dont want them to promote. Now, you might be thinking “I want them to index every page on my site”, and that’s a fair call, but what they can also do, is promote the folders and files that exist behind the scenes of your site, that you would prefer to keep private for reasons such as privacy and security.
If you are new to the web/site design, and this is all news to you, and you have realised there are some folders on your server that contain files you don’t want the world to find through a search engine, you can easily generate a robots.txt file using the free robots.txt file generator at McAnerin HERE. Once you have told the generator the directories on your server you want to keep private, copy the code it generates, and then create a new text document on your computer, and paste it in. Call the file “robots.txt” (without the exclamation marks of course) and then upload it into the “public_html” folder of your website.
Now, for those of us who use WordPress, there are quiet a few directories that I think should be blocked. There are also others who have the opinion that your WordPress site ranks higher if you don’t use one. There’s still a lot of unanswered questions about the internet, and this is one of them. Until it is resolved, I personally will continue to use robots.txt.
So for now, below is my robots.txt file I use – If you think I have missed something, or if you think the whole robots.txt idea is junk, please comment! I’d love to hear your thought’s for and against.
User-agent: *
Disallow: /wp-content/
Disallow: /wp-icludes/
Disallow: /trackback/
Disallow: /wp-admin/
Disallow: /archives/
Disallow: /category/
Disallow: /tag/*
Disallow: /tag/
Disallow: /wp-*
Disallow: /login/
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.php$
User-agent: All
Allow: /
User-agent: Googlebot-Image
Disallow: /
User-agent: ia_archiver
Disallow: /
User-agent: duggmirror
Disallow: /