Robots.txt Nutzungsverteilung In St. Lucia

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Googlebot Disallow	123	9,45
Baidu Bot Disallow	113	8,68
Yandex Bot Disallow	101	7,76
Alexa Bot Disallow	100	7,68
Sogou Bot Disallow	99	7,6
Yahoo Slurp Disallow	98	7,53
Microsoft BingBot Disallow	96	7,37
Soso Bot Disallow	95	7,3
Robots Disallow	74	5,68
Ahrefs Bot Disallow	74	5,68
Google Media Partners Disallow	45 ^*	3,46
Some Robots Disallow	39	3
GPTBot Disallow	25	1,92
Common Crawl Bot Disallow	21	1,61
AppleBot Disallow	21	1,61
Anthropic Claude Bot Disallow	21	1,61
Google Bard Disallow	21	1,61
Facebook Bot Disallow	21	1,61
Apple AI Bot Disallow	21	1,61
Cohere AI Disallow	21	1,61
Perplexity	20	1,54
Google AdsBot Mobile Apps Disallow	20	1,54
Majestic Bot Disallow	6	0,46
Semrush Bot Disallow	5	0,38
ByteDance Bytespider Disallow	3	0,23
Microsoft MSNBot Disallow	3	0,23
Google Image Bot Disallow	2	0,15
Diffbot Disallow	2	0,15
Pipl Disallow	2	0,15
Exalead Bot Disallow	1	0,08
Webz Omgili Disallow	1	0,08
360 Bot Disallow	1	0,08
Google News Bot Disallow	1	0,08
Google Video Bot Disallow	1	0,08
DuckDuckGo Bot Disallow	1	0,08
Google Favicon Disallow	1	0,08
YOU Disallow	1	0,08
Google API Bot Disallow	1	0,08
Google DuplexWeb Disallow	1	0,08