Robots.txt Distribución de uso en los 100.000 mejores sitios

Distribución para sitios web que utilizan Robots.txt tecnologías

Tecnología	Sitios web	%
Robots Disallow	29.711	29,71
Googlebot Disallow	9.783	9,78
Some Robots Disallow	9.322	9,32
Ahrefs Bot Disallow	9.093	9,09
GPTBot Disallow	8.208	8,21
Yandex Bot Disallow	6.593	6,59
Semrush Bot Disallow	6.183	6,18
Common Crawl Bot Disallow	5.820	5,82
Google AdsBot Disallow	5.753	5,75
AppleBot Disallow	5.676	5,68
Majestic Bot Disallow	5.350	5,35
Anthropic Claude Bot Disallow	4.909	4,91
Baidu Bot Disallow	4.785	4,78
Microsoft BingBot Disallow	4.369	4,37
Google Bard Disallow	4.179	4,18
ByteDance Bytespider Disallow	3.481	3,48
Facebook Bot Disallow	3.402	3,4
Perplexity	3.003	3
Alexa Bot Disallow	2.944	2,94
Webz Omgili Disallow	2.904	2,9
Yahoo Slurp Disallow	2.743	2,74
Apple AI Bot Disallow	2.679	2,68
Cohere AI Disallow	2.666	2,67
Google Media Partners Disallow	2.610	2,61
Microsoft MSNBot Disallow	2.502	2,5
Google Image Bot Disallow	2.205	2,2
Sogou Bot Disallow	2.131	2,13
Google AdsBot Mobile Disallow	2.111	2,11
Exalead Bot Disallow	1.677	1,68
Diffbot Disallow	1.659	1,66
Google News Bot Disallow	1.510	1,51
Google AdsBot Mobile Apps Disallow	1.372	1,37
Soso Bot Disallow	1.349	1,35
DuckDuckGo Bot Disallow	1.231	1,23
360 Bot Disallow	1.078	1,08
YOU Disallow	1.031	1,03
Google Video Bot Disallow	1.027	1,03
Google Favicon Disallow	838	0,84
Google API Bot Disallow	833	0,83
Google DuplexWeb Disallow	791	0,79
BuiltWithBot Disallow	563	0,56
Pipl Disallow	458	0,46
Youdao Bot Disallow	202	0,2
Google FeedFetcher Disallow	35	0,03
Google Read Aloud Disallow	17	0,02
Google Web Light Disallow	7	0,01
Google BIG Bench Disallow	3	0