Robots.txt Usage Distribution in Costa Rica

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	3,073	14.78
Google AdsBot Mobile Disallow	2,516	12.1
Googlebot Disallow	1,939	9.33
GPTBot Disallow	1,003	4.83
Common Crawl Bot Disallow	965	4.64
Anthropic Claude Bot Disallow	947	4.56
AppleBot Disallow	945	4.55
Google Bard Disallow	941	4.53
Facebook Bot Disallow	936	4.5
Google AdsBot Mobile Apps Disallow	932	4.48
Apple AI Bot Disallow	928	4.46
Perplexity	925	4.45
Cohere AI Disallow	921	4.43
Ahrefs Bot Disallow	624	3
Robots Disallow	522	2.51
Baidu Bot Disallow	352	1.69
Google Media Partners Disallow	313 ^*	1.51
Some Robots Disallow	270	1.3
Yandex Bot Disallow	262	1.26
Yahoo Slurp Disallow	202	0.97
Alexa Bot Disallow	188	0.9
Sogou Bot Disallow	184	0.89
Microsoft BingBot Disallow	181	0.87
Soso Bot Disallow	158	0.76
Semrush Bot Disallow	132	0.64
Majestic Bot Disallow	109	0.52
Microsoft MSNBot Disallow	68	0.33
Exalead Bot Disallow	55	0.26
ByteDance Bytespider Disallow	34	0.16
Google Image Bot Disallow	31	0.15
Webz Omgili Disallow	27	0.13
DuckDuckGo Bot Disallow	13	0.06
360 Bot Disallow	11	0.05
Google News Bot Disallow	11	0.05
Google Video Bot Disallow	11	0.05
Google Favicon Disallow	11	0.05
Google API Bot Disallow	10	0.05
Google DuplexWeb Disallow	10	0.05
BuiltWithBot Disallow	7	0.03
Diffbot Disallow	6	0.03
Youdao Bot Disallow	6	0.03
YOU Disallow	4	0.02
Pipl Disallow	3	0.01