Robots.txt Usage Distribution in Denmark

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	33,749	11.1
Ahrefs Bot Disallow	25,232	8.3
Googlebot Disallow	24,198	7.96
Google AdsBot Mobile Disallow	21,300	7
Majestic Bot Disallow	15,606	5.13
GPTBot Disallow	11,667	3.84
Common Crawl Bot Disallow	10,356	3.4
AppleBot Disallow	10,135	3.33
Anthropic Claude Bot Disallow	10,133	3.33
Google Bard Disallow	10,001	3.29
Microsoft BingBot Disallow	9,845	3.24
Facebook Bot Disallow	9,713	3.19
Cohere AI Disallow	9,467	3.11
Yahoo Slurp Disallow	9,427	3.1
Perplexity	9,390	3.09
Apple AI Bot Disallow	9,163	3.01
Robots Disallow	8,450	2.78
Google AdsBot Mobile Apps Disallow	8,429	2.77
Yandex Bot Disallow	6,807	2.24
Baidu Bot Disallow	6,588	2.17
Semrush Bot Disallow	6,326	2.08
Some Robots Disallow	6,178	2.03
Google Image Bot Disallow	5,925	1.95
Alexa Bot Disallow	5,144	1.69
Sogou Bot Disallow	4,166	1.37
Soso Bot Disallow	3,545	1.17
Exalead Bot Disallow	1,853	0.61
ByteDance Bytespider Disallow	1,813	0.6
Webz Omgili Disallow	1,806	0.59
Google Media Partners Disallow	1,369 ^*	0.45
YOU Disallow	1,219	0.4
Diffbot Disallow	1,217	0.4
Microsoft MSNBot Disallow	964	0.32
360 Bot Disallow	689	0.23
DuckDuckGo Bot Disallow	490	0.16
Google News Bot Disallow	364	0.12
Google Video Bot Disallow	359	0.12
Google Favicon Disallow	335	0.11
Google API Bot Disallow	300	0.1
Google DuplexWeb Disallow	283	0.09
BuiltWithBot Disallow	116	0.04
Pipl Disallow	50	0.02
Youdao Bot Disallow	10	0
Google FeedFetcher Disallow	2	0