Robots.txt Usage Distribution in China

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	83,640	10.24
Microsoft BingBot Disallow	76,053	9.31
Ahrefs Bot Disallow	58,653	7.18
Baidu Bot Disallow	57,703	7.06
Yahoo Slurp Disallow	53,066	6.49
Alexa Bot Disallow	52,108	54.9
Semrush Bot Disallow	52,075	6.37
Yandex Bot Disallow	51,779	6.34
Sogou Bot Disallow	50,054	6.13
Soso Bot Disallow	42,438	5.19
Robots Disallow	34,638	4.24
Some Robots Disallow	31,104	3.81
Majestic Bot Disallow	30,839	3.77
AppleBot Disallow	29,980	3.67
Google AdsBot Disallow	23,231	2.84
Google AdsBot Mobile Disallow	16,092	1.97
Microsoft MSNBot Disallow	14,838	1.82
Google Image Bot Disallow	11,042	1.35
GPTBot Disallow	7,319	7.71
Anthropic Claude Bot Disallow	6,655	7.01
ByteDance Bytespider Disallow	5,001	5.27
Exalead Bot Disallow	3,870	0.47
360 Bot Disallow	3,568	0.44
Youdao Bot Disallow	2,891	0.35
Common Crawl Bot Disallow	2,830	2.98
Google Bard Disallow	2,210	2.33
Facebook Bot Disallow	2,197	2.31
Google Media Partners Disallow	1,964 ^*	0.24
Cohere AI Disallow	1,695	1.79
Apple AI Bot Disallow	1,684	1.77
Perplexity	1,555	1.64
Google AdsBot Mobile Apps Disallow	1,495	0.18
Webz Omgili Disallow	519	0.55
DuckDuckGo Bot Disallow	403	0.05
Diffbot Disallow	394	0.42
Google News Bot Disallow	234	0.03
Google Video Bot Disallow	219	0.03
BuiltWithBot Disallow	205	0.03
Google Favicon Disallow	197	0.02
Google API Bot Disallow	196	0.02
Google DuplexWeb Disallow	195	0.02
Google FeedFetcher Disallow	155	0.02
YOU Disallow	101	0.11
LLMS Text	27	0
Pipl Disallow	22	0
Google Read Aloud Disallow	4	0
Google Web Light Disallow	2	0