Robots.txt Usage Distribution in France

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	199,554	15.24
Google AdsBot Disallow	197,956	15.12
Google AdsBot Mobile Disallow	148,264	11.32
Robots Disallow	89,099	6.8
Ahrefs Bot Disallow	62,095	4.74
Baidu Bot Disallow	56,374	4.3
Sogou Bot Disallow	45,193	3.45
Yandex Bot Disallow	37,726	2.88
Some Robots Disallow	32,850	2.51
Microsoft BingBot Disallow	32,764	2.5
Yahoo Slurp Disallow	31,417	2.4
Soso Bot Disallow	27,782	2.12
Alexa Bot Disallow	27,528	2.1
Semrush Bot Disallow	24,752	1.89
GPTBot Disallow	24,286	1.85
Majestic Bot Disallow	22,616	1.73
Common Crawl Bot Disallow	21,609	1.65
Anthropic Claude Bot Disallow	20,170	1.54
Google AdsBot Mobile Apps Disallow	18,805	1.44
Google Bard Disallow	18,319	1.4
AppleBot Disallow	18,281	1.4
Facebook Bot Disallow	17,890	1.37
Perplexity	16,370	1.25
Apple AI Bot Disallow	16,057	1.23
Cohere AI Disallow	15,658	1.2
Google Media Partners Disallow	12,675 ^*	0.97
Google Image Bot Disallow	9,057	0.69
Microsoft MSNBot Disallow	8,241	0.63
DuckDuckGo Bot Disallow	6,465	0.49
ByteDance Bytespider Disallow	5,335	0.41
Google News Bot Disallow	5,174	0.4
Youdao Bot Disallow	5,152	0.39
Google Video Bot Disallow	5,122	0.39
Exalead Bot Disallow	5,044	0.39
Webz Omgili Disallow	4,441	0.34
Google Favicon Disallow	4,363	0.33
Google API Bot Disallow	4,327	0.33
Google DuplexWeb Disallow	4,309	0.33
Diffbot Disallow	2,014	0.15
YOU Disallow	1,818	0.14
360 Bot Disallow	1,659	0.13
BuiltWithBot Disallow	584	0.04
Pipl Disallow	417	0.03
Google FeedFetcher Disallow	21	0
Google Read Aloud Disallow	13	0
Google Web Light Disallow	3	0
LLMS Text	1	0