Robots.txt Usage Distribution in Bermuda

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Common Crawl Bot Disallow	175	8.28
GPTBot Disallow	174	8.23
Google Bard Disallow	172	8.14
AppleBot Disallow	170	8.05
Anthropic Claude Bot Disallow	170	8.05
Facebook Bot Disallow	169	8
Apple AI Bot Disallow	169	8
Perplexity	168	7.95
Cohere AI Disallow	165	7.81
Google AdsBot Mobile Apps Disallow	165	7.81
Ahrefs Bot Disallow	105	4.97
Robots Disallow	71	3.36
Baidu Bot Disallow	30	1.42
Majestic Bot Disallow	26	1.23
Some Robots Disallow	24	1.14
Yandex Bot Disallow	22	1.04
Microsoft BingBot Disallow	18	0.85
Semrush Bot Disallow	18	0.85
Yahoo Slurp Disallow	16	0.76
Sogou Bot Disallow	15	0.71
Alexa Bot Disallow	13	0.62
Soso Bot Disallow	12	0.57
Google Image Bot Disallow	11	0.52
Microsoft MSNBot Disallow	8	0.38
ByteDance Bytespider Disallow	6	0.28
Exalead Bot Disallow	6	0.28
Webz Omgili Disallow	5	0.24
360 Bot Disallow	4	0.19
BuiltWithBot Disallow	3	0.14
Google News Bot Disallow	1	0.05
Google Video Bot Disallow	1	0.05
Google Favicon Disallow	1	0.05