Robots.txt Répartition de l'utilisation dans .co

Distribution pour les sites Web utilisant Robots.txt les technologies

Technologie	Sites Internet	%
Googlebot Disallow	126 536	12,3
Baidu Bot Disallow	100 925	9,81
Google AdsBot Disallow	95 208	9,26
Sogou Bot Disallow	59 538	5,79
Some Robots Disallow	56 110	5,46
Yandex Bot Disallow	50 490	4,91
Microsoft BingBot Disallow	50 371	4,9
Yahoo Slurp Disallow	49 470	4,81
Ahrefs Bot Disallow	49 257	4,79
Soso Bot Disallow	49 023	4,77
Google AdsBot Mobile Disallow	46 787	4,55
Alexa Bot Disallow	42 839	4,16
Anthropic Claude Bot Disallow	21 946	2,13
Common Crawl Bot Disallow	21 713	2,11
GPTBot Disallow	21 380	2,08
AppleBot Disallow	21 080	2,05
Google Bard Disallow	20 961	2,04
Facebook Bot Disallow	20 813	2,02
Apple AI Bot Disallow	20 684	2,01
Perplexity	20 093	1,95
Google AdsBot Mobile Apps Disallow	19 746	1,92
Cohere AI Disallow	19 628	1,91
Robots Disallow	15 017	1,46
Google Media Partners Disallow	7 517 ^*	0,73
Majestic Bot Disallow	6 875	0,67
Semrush Bot Disallow	5 785	0,56
Webz Omgili Disallow	1 392	0,14
360 Bot Disallow	1 199	0,12
Exalead Bot Disallow	1 149	0,11
Microsoft MSNBot Disallow	1 090	0,11
ByteDance Bytespider Disallow	1 038	0,1
Google Image Bot Disallow	782	0,08
BuiltWithBot Disallow	521	0,05
Google News Bot Disallow	256	0,02
Google Video Bot Disallow	235	0,02
Google Favicon Disallow	202	0,02
Diffbot Disallow	187	0,02
Youdao Bot Disallow	187	0,02
DuckDuckGo Bot Disallow	173	0,02
Google API Bot Disallow	115	0,01
Google DuplexWeb Disallow	115	0,01
YOU Disallow	79	0,01
Pipl Disallow	74	0,01
Google FeedFetcher Disallow	1	0