Robots.txt Distribución de uso en Reunión

Distribución para sitios web que utilizan Robots.txt tecnologías

Tecnología	Sitios web	%
Googlebot Disallow	2.840	17,98
Google AdsBot Disallow	1.395	8,83
Baidu Bot Disallow	1.393	8,82
Robots Disallow	1.222	7,74
Some Robots Disallow	1.077	6,82
Sogou Bot Disallow	1.056	6,69
Google AdsBot Mobile Disallow	1.044	6,61
Yandex Bot Disallow	601	3,81
Microsoft BingBot Disallow	592	3,75
Ahrefs Bot Disallow	577	3,65
Yahoo Slurp Disallow	574	3,63
Alexa Bot Disallow	526	3,33
Soso Bot Disallow	524	3,32
Semrush Bot Disallow	255	1,61
Google Media Partners Disallow	254 ^*	1,61
Majestic Bot Disallow	211	1,34
Anthropic Claude Bot Disallow	207	1,31
GPTBot Disallow	127	0,8
Common Crawl Bot Disallow	120	0,76
Google Bard Disallow	110	0,7
Facebook Bot Disallow	101	0,64
AppleBot Disallow	96	0,61
Apple AI Bot Disallow	92	0,58
Google AdsBot Mobile Apps Disallow	89	0,56
Google Image Bot Disallow	86	0,54
DuckDuckGo Bot Disallow	74	0,47
Cohere AI Disallow	68	0,43
Perplexity	68	0,43
Microsoft MSNBot Disallow	58	0,37
Webz Omgili Disallow	58	0,37
Google News Bot Disallow	41	0,26
Google Video Bot Disallow	41	0,26
Google Favicon Disallow	39	0,25
Google API Bot Disallow	39	0,25
Google DuplexWeb Disallow	39	0,25
Exalead Bot Disallow	29	0,18
ByteDance Bytespider Disallow	26	0,16
Diffbot Disallow	17	0,11
YOU Disallow	16	0,1
Youdao Bot Disallow	3	0,02
360 Bot Disallow	2	0,01
BuiltWithBot Disallow	2	0,01
Pipl Disallow	2	0,01