Robots.txt Usage Distribution in Trinidad and Tobago

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	172	9.67
Google AdsBot Mobile Disallow	156	8.77
Googlebot Disallow	114	6.41
Robots Disallow	110	6.18
GPTBot Disallow	93	12.14
AppleBot Disallow	93	5.23
Google Bard Disallow	92	12.01
Common Crawl Bot Disallow	91	11.88
Anthropic Claude Bot Disallow	91	11.88
Facebook Bot Disallow	91	11.88
Apple AI Bot Disallow	91	11.88
Cohere AI Disallow	89	11.62
Perplexity	89	11.62
Google AdsBot Mobile Apps Disallow	89	5
Baidu Bot Disallow	35	1.97
Yahoo Slurp Disallow	31	1.74
Sogou Bot Disallow	29	1.63
Yandex Bot Disallow	29	1.63
Microsoft BingBot Disallow	26	1.46
Some Robots Disallow	25	1.41
Alexa Bot Disallow	25	3.26
Ahrefs Bot Disallow	25	1.41
Soso Bot Disallow	21	1.18
Google Media Partners Disallow	13 ^*	0.73
Majestic Bot Disallow	8	0.45
Semrush Bot Disallow	8	0.45
Microsoft MSNBot Disallow	8	0.45
Google Image Bot Disallow	8	0.45
Webz Omgili Disallow	5	0.65
Exalead Bot Disallow	3	0.17
Google News Bot Disallow	3	0.17
Google Video Bot Disallow	3	0.17
DuckDuckGo Bot Disallow	3	0.17
Google Favicon Disallow	3	0.17
ByteDance Bytespider Disallow	2	0.26
360 Bot Disallow	1	0.06
Diffbot Disallow	1	0.13
YOU Disallow	1	0.13
Google API Bot Disallow	1	0.06
Google DuplexWeb Disallow	1	0.06