Robots.txt Usage Distribution in the Top 1 Million Sites

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Robots Disallow	149,330	14.93
Ahrefs Bot Disallow	60,592	6.06
Googlebot Disallow	59,594	5.96
Yandex Bot Disallow	47,428	4.74
Google AdsBot Disallow	41,199	4.12
Some Robots Disallow	37,748	3.77
Baidu Bot Disallow	36,588	3.66
Semrush Bot Disallow	30,239	3.02
Majestic Bot Disallow	30,091	3.01
Microsoft BingBot Disallow	27,481	2.75
GPTBot Disallow	26,838	2.68
Google Media Partners Disallow	24,371	2.44
Yahoo Slurp Disallow	22,656	2.27
Alexa Bot Disallow	22,564	2.26
Common Crawl Bot Disallow	19,714	1.97
Sogou Bot Disallow	17,731	1.77
Soso Bot Disallow	15,017	1.5
Anthropic Claude Bot Disallow	13,928	1.39
Microsoft MSNBot Disallow	12,548	1.25
Google Bard Disallow	12,058	1.21
Google AdsBot Mobile Disallow	11,382	1.14
Google Image Bot Disallow	10,464	1.05
AppleBot Disallow	9,472	0.95
Facebook Bot Disallow	8,886	0.89
Exalead Bot Disallow	8,732	0.87
ByteDance Bytespider Disallow	7,946	0.79
Google AdsBot Mobile Apps Disallow	7,449	0.74
Perplexity	6,883	0.69
Google News Bot Disallow	6,649	0.66
Webz Omgili Disallow	5,795	0.58
Cohere AI Disallow	5,784	0.58
Apple AI Bot Disallow	5,564	0.56
DuckDuckGo Bot Disallow	5,510	0.55
Google Video Bot Disallow	5,180	0.52
360 Bot Disallow	5,110	0.51
Google Favicon Disallow	4,544	0.45
Google API Bot Disallow	4,447	0.44
Google DuplexWeb Disallow	4,346	0.43
Diffbot Disallow	2,501	0.25
BuiltWithBot Disallow	2,267	0.23
Pipl Disallow	1,875	0.19
YOU Disallow	1,588	0.16
Youdao Bot Disallow	910	0.09
Google FeedFetcher Disallow	134	0.01
Google Read Aloud Disallow	47	0
Google Web Light Disallow	16	0
Google BIG Bench Disallow	3	0