Robots.txt Usage Distribution in Isle of Man

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	2,082	10.34
Baidu Bot Disallow	2,008	9.97
Sogou Bot Disallow	1,930	9.59
Yandex Bot Disallow	1,702	8.45
Microsoft BingBot Disallow	1,260	6.26
Alexa Bot Disallow	1,202	5.97
Yahoo Slurp Disallow	1,197	5.95
Soso Bot Disallow	1,190	5.91
Some Robots Disallow	1,053	5.23
Robots Disallow	767	3.81
Google AdsBot Disallow	766	3.8
Ahrefs Bot Disallow	740	3.68
Anthropic Claude Bot Disallow	589	2.93
Semrush Bot Disallow	565	2.81
Majestic Bot Disallow	558	2.77
GPTBot Disallow	536	2.66
Common Crawl Bot Disallow	509	2.53
Google Bard Disallow	497	2.47
Google Media Partners Disallow	133 ^*	0.66
Google AdsBot Mobile Disallow	114	0.57
Facebook Bot Disallow	97	0.48
AppleBot Disallow	92	0.46
Apple AI Bot Disallow	88	0.44
Cohere AI Disallow	63	0.31
Perplexity	60	0.3
Webz Omgili Disallow	55	0.27
Google AdsBot Mobile Apps Disallow	48	0.24
ByteDance Bytespider Disallow	35	0.17
Microsoft MSNBot Disallow	27	0.13
Exalead Bot Disallow	26	0.13
360 Bot Disallow	25	0.12
Diffbot Disallow	20	0.1
Google Image Bot Disallow	17	0.08
YOU Disallow	15	0.07
DuckDuckGo Bot Disallow	10	0.05
Google News Bot Disallow	8	0.04
Google Video Bot Disallow	8	0.04
Google Favicon Disallow	8	0.04
Google API Bot Disallow	8	0.04
Google DuplexWeb Disallow	8	0.04
Pipl Disallow	7	0.03
BuiltWithBot Disallow	4	0.02
Google FeedFetcher Disallow	4	0.02
LLMS Text	1	0