sonofbluerobot
J. Albert Bowden II
Joined 20 Nov 2007
Links 1 through 10 of 16 by J. Albert Bowden II tagged robots.txt
http://www.google.com/support/webmasters/bin/answer.py?answer=79892
This link recently saved by sonofbluerobot on May 18, 2011
1 Save
http://sixrevisions.com/web-standards/5-web-files-that-will-improve-your-we...
This link recently saved by sonofbluerobot on March 16, 2011
890 Saves
http://www.google.com/support/webmasters/bin/answer.py?answer=93710
This link recently saved by sonofbluerobot on December 18, 2010
21 Saves
To entirely prevent a page's contents from being listed in the Google web index even if other sites link to it, use a noindex meta tag. As long as Googlebot fetches the page, it will see the noindex meta tag and prevent that page from showing up in the web index.
The noindex meta standard is described at http://www.robotstxt.org/meta.html. This method is useful if you don't have root access to your server, as it allows you to control access to your site on a page-by-page basis.
To prevent all robots from indexing a page on your site, place the following meta tag into the <head> section of your page:
<meta name="robots" content="noindex">
To allow other robots to index the page on your site, preventing only Google's robots from indexing the page:
<meta name="googlebot" content="noindex">
http://www.google.com/support/webmasters/bin/answer.py?answer=182072
This link recently saved by sonofbluerobot on December 18, 2010
24 Saves
http://feedblog.org/2010/12/14/the-robot-exclusion-protocol-rep-is-stating-...
This link recently saved by sonofbluerobot on December 18, 2010
1 Save
http://www.robotstxt.org/meta.html
This link recently saved by sonofbluerobot on August 03, 2010
140 Saves
http://davidwalsh.name/robots-txt
This link recently saved by sonofbluerobot on July 26, 2010
14 Saves
This link recently saved by sonofbluerobot on June 16, 2010
1262 Saves
http://searchengineland.com/yahoo-google-microsoft-clarify-robotstxt-suppor...
This link recently saved by sonofbluerobot on June 15, 2010
1 Save
http://www.perldesignpatterns.com/robots.txt
This link recently saved by sonofbluerobot on June 04, 2010
1 Save