Robots.txt Nutzungsverteilung In Isle of Man

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Yandex Bot Disallow	2.326	11
Googlebot Disallow	2.072	9,8
Baidu Bot Disallow	1.985	9,39
Sogou Bot Disallow	1.510	7,14
Microsoft BingBot Disallow	1.392	6,58
Some Robots Disallow	1.379	6,52
Soso Bot Disallow	1.341	6,34
Alexa Bot Disallow	1.340	6,34
Yahoo Slurp Disallow	1.332	6,3
Anthropic Claude Bot Disallow	1.084	5,13
Ahrefs Bot Disallow	817	3,86
Google AdsBot Disallow	814	3,85
Semrush Bot Disallow	718	3,4
Majestic Bot Disallow	713	3,37
Robots Disallow	697	3,3
Google AdsBot Mobile Disallow	161	0,76
GPTBot Disallow	157	0,74
Google Media Partners Disallow	152 ^*	0,72
Common Crawl Bot Disallow	135	0,64
Google Bard Disallow	129	0,61
Facebook Bot Disallow	123	0,58
AppleBot Disallow	120	0,57
Apple AI Bot Disallow	113	0,53
Cohere AI Disallow	82	0,39
Perplexity	79	0,37
Google AdsBot Mobile Apps Disallow	72	0,34
Webz Omgili Disallow	59	0,28
ByteDance Bytespider Disallow	33	0,16
Microsoft MSNBot Disallow	32	0,15
Exalead Bot Disallow	28	0,13
360 Bot Disallow	21	0,1
Google Image Bot Disallow	20	0,09
Diffbot Disallow	16	0,08
YOU Disallow	11	0,05
DuckDuckGo Bot Disallow	11	0,05
Google News Bot Disallow	10	0,05
Google Video Bot Disallow	10	0,05
Google Favicon Disallow	10	0,05
Google API Bot Disallow	10	0,05
Google DuplexWeb Disallow	10	0,05
Pipl Disallow	8	0,04
BuiltWithBot Disallow	7	0,03
Google FeedFetcher Disallow	2	0,01
Youdao Bot Disallow	1	0