Robots.txt Usage Distribution in Turks and Caicos Islands

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	100	7.95
Googlebot Disallow	98	7.79
Robots Disallow	68	5.41
GPTBot Disallow	65	5.17
Common Crawl Bot Disallow	64	5.09
Google Bard Disallow	62	4.93
Anthropic Claude Bot Disallow	61	4.85
Facebook Bot Disallow	61	4.85
AppleBot Disallow	60	4.77
Apple AI Bot Disallow	60	4.77
Google AdsBot Mobile Apps Disallow	60	4.77
Google Media Partners Disallow	60 ^*	4.77
Perplexity	59	4.69
Cohere AI Disallow	58	4.61
Baidu Bot Disallow	57	4.53
Some Robots Disallow	51	4.05
Yandex Bot Disallow	30	2.38
Sogou Bot Disallow	29	2.31
Yahoo Slurp Disallow	25	1.99
Alexa Bot Disallow	24	1.91
Microsoft BingBot Disallow	22	1.75
Soso Bot Disallow	21	1.67
Ahrefs Bot Disallow	14	1.11
Microsoft MSNBot Disallow	8	0.64
Majestic Bot Disallow	7	0.56
Semrush Bot Disallow	5	0.4
ByteDance Bytespider Disallow	3	0.24
Exalead Bot Disallow	3	0.24
Webz Omgili Disallow	3	0.24
Google Image Bot Disallow	3	0.24
360 Bot Disallow	3	0.24
BuiltWithBot Disallow	2	0.16
Google News Bot Disallow	2	0.16
Google Video Bot Disallow	2	0.16
DuckDuckGo Bot Disallow	2	0.16
Google Favicon Disallow	2	0.16
Google API Bot Disallow	2	0.16
Google DuplexWeb Disallow	2	0.16