Robots.txt Distribución de uso en los 10.000 mejores sitios

Distribución para sitios web que utilizan Robots.txt tecnologías

Tecnología	Sitios web	%
Robots Disallow	3.689	36,89
Some Robots Disallow	1.488	14,88
GPTBot Disallow	1.474	14,74
Googlebot Disallow	1.374	13,74
Common Crawl Bot Disallow	1.139	11,39
AppleBot Disallow	1.050	10,5
Ahrefs Bot Disallow	1.018	10,18
Anthropic Claude Bot Disallow	999	9,99
Google AdsBot Disallow	871	8,71
Yandex Bot Disallow	783	7,83
ByteDance Bytespider Disallow	776	7,76
Semrush Bot Disallow	734	7,34
Google Bard Disallow	722	7,22
Facebook Bot Disallow	721	7,21
Perplexity	709	7,09
Webz Omgili Disallow	683	6,83
Cohere AI Disallow	650	6,5
Apple AI Bot Disallow	638	6,38
Microsoft BingBot Disallow	599	5,99
Majestic Bot Disallow	594	5,94
Baidu Bot Disallow	592	5,92
Diffbot Disallow	470	4,7
Alexa Bot Disallow	432	4,32
Yahoo Slurp Disallow	341	3,41
Google Media Partners Disallow	324	3,24
Microsoft MSNBot Disallow	320	3,2
Google AdsBot Mobile Disallow	295	2,95
Sogou Bot Disallow	278	2,78
Google Image Bot Disallow	266	2,66
Exalead Bot Disallow	206	2,06
YOU Disallow	199	1,99
Google News Bot Disallow	189	1,89
Google AdsBot Mobile Apps Disallow	179	1,79
Soso Bot Disallow	168	1,68
DuckDuckGo Bot Disallow	147	1,47
360 Bot Disallow	137	1,37
Google Video Bot Disallow	114	1,14
Google API Bot Disallow	83	0,83
Google Favicon Disallow	78	0,78
Google DuplexWeb Disallow	74	0,74
Pipl Disallow	66	0,66
BuiltWithBot Disallow	62	0,62
Youdao Bot Disallow	25	0,25
Google FeedFetcher Disallow	3	0,03
Google Web Light Disallow	1	0,01
Google BIG Bench Disallow	1	0,01