sonofbluerobot
J. Albert Bowden II
Joined 20 Nov 2007
Links 1 through 10 of 18 by J. Albert Bowden II tagged bots
http://planb.nicecupoftea.org/2012/01/05/a-node-js-bot-in-xmpp/
This link recently saved by sonofbluerobot on January 19, 2012
1 Save
http://www.google.com/support/webmasters/bin/answer.py?answer=79892
This link recently saved by sonofbluerobot on May 18, 2011
1 Save
http://searchengineland.com/yahoo-provides-noydir-opt-out-of-yahoo-director...
This link recently saved by sonofbluerobot on May 18, 2011
1 Save
http://searchengineland.com/microsoft-to-replace-msnbot-with-bingbot-octobe...
This link recently saved by sonofbluerobot on December 18, 2010
1 Save
Mozilla/5.0 (compatible; bingbot/2.0 +http://www.bing.com/bingbot.htm)
Note, if you already have directives for MSNBot in your robots.txt file, BingBot will obey those directives. It is important to note that if you have two different directives, one for MSNBot and one for BingBot and they say two different things, Microsoft will listen to BingBot over MSNBot.
http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=178852
This link recently saved by sonofbluerobot on December 18, 2010
1 Save
http://www.google.com/support/webmasters/bin/answer.py?answer=158587
This link recently saved by sonofbluerobot on December 18, 2010
19 Saves
http://www.google.com/support/webmasters/bin/answer.py?answer=93710
This link recently saved by sonofbluerobot on December 18, 2010
21 Saves
To entirely prevent a page's contents from being listed in the Google web index even if other sites link to it, use a noindex meta tag. As long as Googlebot fetches the page, it will see the noindex meta tag and prevent that page from showing up in the web index.
The noindex meta standard is described at http://www.robotstxt.org/meta.html. This method is useful if you don't have root access to your server, as it allows you to control access to your site on a page-by-page basis.
To prevent all robots from indexing a page on your site, place the following meta tag into the <head> section of your page:
<meta name="robots" content="noindex">
To allow other robots to index the page on your site, preventing only Google's robots from indexing the page:
<meta name="googlebot" content="noindex">
http://www.google.com/support/webmasters/bin/answer.py?answer=182072
This link recently saved by sonofbluerobot on December 18, 2010
24 Saves
This link recently saved by sonofbluerobot on December 02, 2010
30 Saves
http://googlewebmastercentral.blogspot.com/2008/04/crawling-through-html-fo...
googlewebmastercentral.blogspot.com
This link recently saved by sonofbluerobot on November 20, 2010
99 Saves