Robots.txt Usage Distribution in Andorra

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	220	13.5
Google AdsBot Disallow	184	11.29
Google AdsBot Mobile Disallow	157	9.63
Robots Disallow	118	7.24
Some Robots Disallow	112	6.87
Baidu Bot Disallow	68	4.17
Ahrefs Bot Disallow	61	3.74
Yandex Bot Disallow	59	3.62
Alexa Bot Disallow	54	3.31
Yahoo Slurp Disallow	49	3.01
GPTBot Disallow	42	2.58
Microsoft BingBot Disallow	41	2.52
Soso Bot Disallow	39	2.39
Anthropic Claude Bot Disallow	38	2.33
Common Crawl Bot Disallow	35	2.15
AppleBot Disallow	35	2.15
Google Bard Disallow	34	2.09
Facebook Bot Disallow	34	2.09
Apple AI Bot Disallow	34	2.09
Perplexity	34	2.09
Google AdsBot Mobile Apps Disallow	34	2.09
Cohere AI Disallow	33	2.02
Majestic Bot Disallow	29	1.78
Semrush Bot Disallow	27	1.66
Google Media Partners Disallow	17 ^*	1.04
Microsoft MSNBot Disallow	12	0.74
ByteDance Bytespider Disallow	7	0.43
Google Image Bot Disallow	5	0.31
Exalead Bot Disallow	4	0.25
Youdao Bot Disallow	4	0.25
Webz Omgili Disallow	2	0.12
Diffbot Disallow	1	0.06
Google News Bot Disallow	1	0.06
Google Video Bot Disallow	1	0.06
YOU Disallow	1	0.06
DuckDuckGo Bot Disallow	1	0.06
Google Favicon Disallow	1	0.06
Google API Bot Disallow	1	0.06
Google DuplexWeb Disallow	1	0.06