Robots.txt Nutzungsverteilung in St. Lucia

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Googlebot Disallow	159	9,96
Baidu Bot Disallow	156	9,77
Sogou Bot Disallow	152	9,52
Yandex Bot Disallow	128	8,02
Yahoo Slurp Disallow	127	7,96
Microsoft BingBot Disallow	126	7,89
Alexa Bot Disallow	126	7,89
Soso Bot Disallow	124	7,77
Ahrefs Bot Disallow	123	7,71
Robots Disallow	89	5,58
Google Media Partners Disallow	53 ^*	3,32
Some Robots Disallow	32	2,01
GPTBot Disallow	21	1,32
AppleBot Disallow	17	1,07
Common Crawl Bot Disallow	16	1
Anthropic Claude Bot Disallow	16	1
Google Bard Disallow	16	1
Facebook Bot Disallow	16	1
Cohere AI Disallow	16	1
Apple AI Bot Disallow	16	1
Perplexity	16	1
Google AdsBot Mobile Apps Disallow	16	1
360 Bot Disallow	7	0,44
Majestic Bot Disallow	5	0,31
Semrush Bot Disallow	5	0,31
Microsoft MSNBot Disallow	4	0,25
ByteDance Bytespider Disallow	2	0,13
Google Image Bot Disallow	2	0,13
Webz Omgili Disallow	1	0,06
Diffbot Disallow	1	0,06
YOU Disallow	1	0,06
Google News Bot Disallow	1	0,06
Google Video Bot Disallow	1	0,06
DuckDuckGo Bot Disallow	1	0,06
Google Favicon Disallow	1	0,06
Youdao Bot Disallow	1	0,06
Google API Bot Disallow	1	0,06
Google DuplexWeb Disallow	1	0,06