Robots.txt Usage Distribution in the Top 10k Sites

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Robots Disallow	3,633	36.33
GPTBot Disallow	1,481	14.81
Some Robots Disallow	1,466	14.66
Googlebot Disallow	1,383	13.83
Common Crawl Bot Disallow	1,157	11.57
AppleBot Disallow	1,117	11.17
Anthropic Claude Bot Disallow	1,036	10.36
Ahrefs Bot Disallow	1,002	10.02
Google AdsBot Disallow	835	8.35
ByteDance Bytespider Disallow	834	8.34
Yandex Bot Disallow	777	7.77
Google Bard Disallow	764	7.64
Perplexity	759	7.59
Facebook Bot Disallow	747	7.47
Semrush Bot Disallow	742	7.42
Webz Omgili Disallow	702	7.02
Cohere AI Disallow	684	6.84
Apple AI Bot Disallow	656	6.56
Majestic Bot Disallow	599	5.99
Microsoft BingBot Disallow	598	5.98
Baidu Bot Disallow	575	5.75
Diffbot Disallow	506	5.06
Alexa Bot Disallow	432	4.32
Yahoo Slurp Disallow	337	3.37
Google Media Partners Disallow	324	3.24
Microsoft MSNBot Disallow	319	3.19
Google AdsBot Mobile Disallow	285	2.85
Sogou Bot Disallow	274	2.74
Google Image Bot Disallow	260	2.6
YOU Disallow	251	2.51
Exalead Bot Disallow	209	2.09
Google News Bot Disallow	189	1.89
Soso Bot Disallow	169	1.69
Google AdsBot Mobile Apps Disallow	168	1.68
DuckDuckGo Bot Disallow	144	1.44
360 Bot Disallow	141	1.41
Google Video Bot Disallow	110	1.1
Google API Bot Disallow	75	0.75
Google Favicon Disallow	72	0.72
Pipl Disallow	72	0.72
Google DuplexWeb Disallow	68	0.68
BuiltWithBot Disallow	64	0.64
Youdao Bot Disallow	25	0.25
LLMS Text	17	0.17
Google FeedFetcher Disallow	4	0.04
Google Read Aloud Disallow	1	0.01
Google Web Light Disallow	1	0.01
Google BIG Bench Disallow	1	0.01