Robots.txt Usage Distribution in Åland Islands

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	96	21.87
AppleBot Disallow	29	6.61
Anthropic Claude Bot Disallow	28	6.38
GPTBot Disallow	27	6.15
Common Crawl Bot Disallow	26	5.92
Google Bard Disallow	26	5.92
Facebook Bot Disallow	26	5.92
Apple AI Bot Disallow	26	5.92
Perplexity	26	5.92
Cohere AI Disallow	25	5.69
Google AdsBot Mobile Apps Disallow	22	5.01
Robots Disallow	17	3.87
Yandex Bot Disallow	9	2.05
Ahrefs Bot Disallow	7	1.59
Majestic Bot Disallow	5	1.14
ByteDance Bytespider Disallow	5	1.14
Webz Omgili Disallow	5	1.14
Semrush Bot Disallow	4	0.91
Diffbot Disallow	4	0.91
YOU Disallow	4	0.91
Microsoft BingBot Disallow	3	0.68
Alexa Bot Disallow	3	0.68
Yahoo Slurp Disallow	3	0.68
DuckDuckGo Bot Disallow	3	0.68
Soso Bot Disallow	2	0.46
Microsoft MSNBot Disallow	2	0.46
Google Image Bot Disallow	1	0.23
Google News Bot Disallow	1	0.23
Google Video Bot Disallow	1	0.23
Google Favicon Disallow	1	0.23
Google API Bot Disallow	1	0.23
Google DuplexWeb Disallow	1	0.23