Robots.txt Usage Distribution in Puerto Rico

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	4,301	22.38
Google AdsBot Mobile Disallow	2,487	12.94
Googlebot Disallow	2,443	12.71
Ahrefs Bot Disallow	1,095	5.7
AppleBot Disallow	827	4.3
Baidu Bot Disallow	591	3.08
Yandex Bot Disallow	508	2.64
Common Crawl Bot Disallow	502	2.61
GPTBot Disallow	500	2.6
Anthropic Claude Bot Disallow	493	2.57
Google Bard Disallow	489	2.54
Facebook Bot Disallow	488	2.54
Apple AI Bot Disallow	486	2.53
Cohere AI Disallow	479	2.49
Perplexity	477	2.48
Google AdsBot Mobile Apps Disallow	471	2.45
Microsoft BingBot Disallow	463	2.41
Yahoo Slurp Disallow	455	2.37
ByteDance Bytespider Disallow	352	1.83
Google Image Bot Disallow	349	1.82
Robots Disallow	227	1.18
Sogou Bot Disallow	139	0.72
Alexa Bot Disallow	121	0.63
Some Robots Disallow	116	0.6
Soso Bot Disallow	108	0.56
Majestic Bot Disallow	65	0.34
Semrush Bot Disallow	55	0.29
Google Media Partners Disallow	44 ^*	0.23
Webz Omgili Disallow	23	0.12
Microsoft MSNBot Disallow	22	0.11
Exalead Bot Disallow	13	0.07
YOU Disallow	8	0.04
360 Bot Disallow	4	0.02
Diffbot Disallow	4	0.02
BuiltWithBot Disallow	3	0.02
DuckDuckGo Bot Disallow	3	0.02
Google News Bot Disallow	1	0.01
Google Video Bot Disallow	1	0.01
Google Favicon Disallow	1	0.01
Google API Bot Disallow	1	0.01
Google DuplexWeb Disallow	1	0.01
Pipl Disallow	1	0.01