Robots.txt Usage Distribution in .co

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	126,536	12.3
Baidu Bot Disallow	100,925	9.81
Google AdsBot Disallow	95,208	9.26
Sogou Bot Disallow	59,538	5.79
Some Robots Disallow	56,110	5.46
Yandex Bot Disallow	50,490	4.91
Microsoft BingBot Disallow	50,371	4.9
Yahoo Slurp Disallow	49,470	4.81
Ahrefs Bot Disallow	49,257	4.79
Soso Bot Disallow	49,023	4.77
Google AdsBot Mobile Disallow	46,787	4.55
Alexa Bot Disallow	42,839	4.16
Anthropic Claude Bot Disallow	21,946	2.13
Common Crawl Bot Disallow	21,713	2.11
GPTBot Disallow	21,380	2.08
AppleBot Disallow	21,080	2.05
Google Bard Disallow	20,961	2.04
Facebook Bot Disallow	20,813	2.02
Apple AI Bot Disallow	20,684	2.01
Perplexity	20,093	1.95
Google AdsBot Mobile Apps Disallow	19,746	1.92
Cohere AI Disallow	19,628	1.91
Robots Disallow	15,017	1.46
Google Media Partners Disallow	7,517 ^*	0.73
Majestic Bot Disallow	6,875	0.67
Semrush Bot Disallow	5,785	0.56
Webz Omgili Disallow	1,392	0.14
360 Bot Disallow	1,199	0.12
Exalead Bot Disallow	1,149	0.11
Microsoft MSNBot Disallow	1,090	0.11
ByteDance Bytespider Disallow	1,038	0.1
Google Image Bot Disallow	782	0.08
BuiltWithBot Disallow	521	0.05
Google News Bot Disallow	256	0.02
Google Video Bot Disallow	235	0.02
Google Favicon Disallow	202	0.02
Diffbot Disallow	187	0.02
Youdao Bot Disallow	187	0.02
DuckDuckGo Bot Disallow	173	0.02
Google API Bot Disallow	115	0.01
Google DuplexWeb Disallow	115	0.01
YOU Disallow	79	0.01
Pipl Disallow	74	0.01
Google FeedFetcher Disallow	1	0