Robots.txt Usage Distribution in France

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	196,776	14.95
Google AdsBot Disallow	193,088	14.67
Google AdsBot Mobile Disallow	147,143	11.18
Robots Disallow	73,846	5.61
Ahrefs Bot Disallow	65,843	5
Baidu Bot Disallow	53,412	4.06
Yandex Bot Disallow	37,212	2.83
Sogou Bot Disallow	33,249	2.53
Some Robots Disallow	31,976	2.43
Microsoft BingBot Disallow	31,185	2.37
Yahoo Slurp Disallow	29,688	2.26
Semrush Bot Disallow	29,125	2.21
GPTBot Disallow	27,844	2.12
Majestic Bot Disallow	26,939	2.05
Soso Bot Disallow	26,632	2.02
Alexa Bot Disallow	25,282	1.92
Common Crawl Bot Disallow	24,857	1.89
Anthropic Claude Bot Disallow	23,769	1.81
Google Bard Disallow	21,700	1.65
Google AdsBot Mobile Apps Disallow	21,668	1.65
AppleBot Disallow	21,354	1.62
Facebook Bot Disallow	21,046	1.6
Apple AI Bot Disallow	19,151	1.45
Perplexity	19,125	1.45
Cohere AI Disallow	17,277	1.31
Google Image Bot Disallow	12,163	0.92
Google Media Partners Disallow	12,087 ^*	0.92
Microsoft MSNBot Disallow	9,602	0.73
DuckDuckGo Bot Disallow	7,059	0.54
ByteDance Bytespider Disallow	6,805	0.52
Youdao Bot Disallow	6,750	0.51
Exalead Bot Disallow	6,146	0.47
Google News Bot Disallow	5,631	0.43
Google Video Bot Disallow	5,594	0.42
Webz Omgili Disallow	5,351	0.41
Google Favicon Disallow	4,644	0.35
Google API Bot Disallow	4,593	0.35
Google DuplexWeb Disallow	4,573	0.35
360 Bot Disallow	2,365	0.18
Diffbot Disallow	1,392	0.11
YOU Disallow	1,197	0.09
BuiltWithBot Disallow	622	0.05
Pipl Disallow	505	0.04
Google FeedFetcher Disallow	26	0
Google Read Aloud Disallow	11	0
Google Web Light Disallow	3	0