Robots.txt Nutzungsverteilung in den Vereinigten Staaten

Verteilung für Websites mit Robots.txt Technologien

Oben In Robots.txt Nutzungsverteilung in den Vereinigten Staaten

Technologie	Websites	%
Yandex Bot Disallow	2.224.474	5,28
Google AdsBot Disallow	2.134.724	5,07
Microsoft BingBot Disallow	2.115.633	5,02
Alexa Bot Disallow	2.072.637	4,92
Robots Disallow	2.027.407	4,81
Ahrefs Bot Disallow	1.992.552	4,73
Yahoo Slurp Disallow	1.990.533	4,72
Common Crawl Bot Disallow	1.938.299	4,6
GPTBot Disallow	1.936.431	4,6
Soso Bot Disallow	1.897.732	4,5
AppleBot Disallow	1.812.225	4,3
Anthropic Claude Bot Disallow	1.811.450	4,3
Google Bard Disallow	1.771.447	4,2
Facebook Bot Disallow	1.754.190	4,16
Apple AI Bot Disallow	1.726.771	4,1
Google AdsBot Mobile Disallow	1.706.207	4,05
Cohere AI Disallow	1.683.613	4
Perplexity	1.658.805	3,94
Google AdsBot Mobile Apps Disallow	1.620.377	3,85
Googlebot Disallow	1.441.905	3,42
Majestic Bot Disallow	854.721	2,03
Semrush Bot Disallow	746.469	1,77
Baidu Bot Disallow	627.758	1,49
Some Robots Disallow	557.025	1,32
Sogou Bot Disallow	291.641	0,69
Google Media Partners Disallow	242.313 ^*	0,58
ByteDance Bytespider Disallow	233.831	0,56
Microsoft MSNBot Disallow	198.093	0,47
Exalead Bot Disallow	195.221	0,46
360 Bot Disallow	176.244	0,42
Google Image Bot Disallow	158.297	0,38
Webz Omgili Disallow	147.512	0,35
Diffbot Disallow	74.196	0,18
BuiltWithBot Disallow	73.060	0,17
Youdao Bot Disallow	42.385	0,1
Google Video Bot Disallow	38.318	0,09
Google News Bot Disallow	38.145	0,09
Google Favicon Disallow	29.846	0,07
DuckDuckGo Bot Disallow	29.726	0,07
YOU Disallow	25.356	0,06
Google API Bot Disallow	10.088	0,02
Google DuplexWeb Disallow	9.721	0,02
Pipl Disallow	8.994	0,02
Google FeedFetcher Disallow	1.770	0
Google Read Aloud Disallow	227	0
LLMS Text	110	0
Google Web Light Disallow	99	0
Google BIG Bench Disallow	15	0