Robots.txt Nutzungsverteilung In Samoa

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Googlebot Disallow	5.349	12,17
Baidu Bot Disallow	5.279	12,01
Sogou Bot Disallow	4.578	10,41
Alexa Bot Disallow	4.490	10,21
Yandex Bot Disallow	4.489	10,21
Microsoft BingBot Disallow	4.455	10,13
Soso Bot Disallow	4.446	10,11
Yahoo Slurp Disallow	4.445	10,11
Some Robots Disallow	3.293	7,49
Google Media Partners Disallow	896 ^*	2,04
Robots Disallow	801	1,82
Ahrefs Bot Disallow	139	0,32
Google AdsBot Disallow	136	0,31
Majestic Bot Disallow	101	0,23
Google AdsBot Mobile Disallow	89	0,2
Common Crawl Bot Disallow	78	0,18
Semrush Bot Disallow	77	0,18
GPTBot Disallow	66	0,15
Microsoft MSNBot Disallow	61	0,14
Anthropic Claude Bot Disallow	58	0,13
AppleBot Disallow	56	0,13
Google Bard Disallow	55	0,13
Facebook Bot Disallow	52	0,12
Apple AI Bot Disallow	48	0,11
Google Image Bot Disallow	47	0,11
Exalead Bot Disallow	46	0,1
Google AdsBot Mobile Apps Disallow	37	0,08
Webz Omgili Disallow	33	0,08
360 Bot Disallow	33	0,08
Google News Bot Disallow	26	0,06
Google Video Bot Disallow	26	0,06
Cohere AI Disallow	25	0,06
Perplexity	25	0,06
Google Favicon Disallow	25	0,06
DuckDuckGo Bot Disallow	20	0,05
Google API Bot Disallow	18	0,04
Google DuplexWeb Disallow	18	0,04
ByteDance Bytespider Disallow	14	0,03
BuiltWithBot Disallow	12	0,03
Diffbot Disallow	9	0,02
YOU Disallow	3	0,01
Youdao Bot Disallow	1	0
Pipl Disallow	1	0
Google FeedFetcher Disallow	1	0