Robots.txt Usage Distribution in the Top 100k Sites

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Robots Disallow	27,186	27.19
Googlebot Disallow	9,034	9.03
Some Robots Disallow	8,372	8.37
Ahrefs Bot Disallow	8,262	8.26
GPTBot Disallow	7,642	7.64
Yandex Bot Disallow	6,140	6.14
Semrush Bot Disallow	5,763	5.76
Common Crawl Bot Disallow	5,270	5.27
Google AdsBot Disallow	5,089	5.09
Majestic Bot Disallow	4,879	4.88
Baidu Bot Disallow	4,447	4.45
Anthropic Claude Bot Disallow	4,257	4.26
AppleBot Disallow	4,131	4.13
Microsoft BingBot Disallow	3,942	3.94
Google Bard Disallow	3,809	3.81
Facebook Bot Disallow	2,951	2.95
ByteDance Bytespider Disallow	2,903	2.9
Alexa Bot Disallow	2,748	2.75
Perplexity	2,561	2.56
Yahoo Slurp Disallow	2,538	2.54
Webz Omgili Disallow	2,523	2.52
Apple AI Bot Disallow	2,355	2.36
Google Media Partners Disallow	2,354	2.35
Microsoft MSNBot Disallow	2,252	2.25
Cohere AI Disallow	2,226	2.23
Sogou Bot Disallow	1,981	1.98
Google Image Bot Disallow	1,949	1.95
Google AdsBot Mobile Disallow	1,841	1.84
Exalead Bot Disallow	1,543	1.54
Google News Bot Disallow	1,334	1.33
Diffbot Disallow	1,319	1.32
Soso Bot Disallow	1,301	1.3
Google AdsBot Mobile Apps Disallow	1,193	1.19
DuckDuckGo Bot Disallow	1,056	1.06
360 Bot Disallow	986	0.99
Google Video Bot Disallow	872	0.87
YOU Disallow	828	0.83
Google Favicon Disallow	700	0.7
Google API Bot Disallow	688	0.69
Google DuplexWeb Disallow	652	0.65
BuiltWithBot Disallow	504	0.5
Pipl Disallow	408	0.41
Youdao Bot Disallow	192	0.19
Google FeedFetcher Disallow	31	0.03
Google Read Aloud Disallow	17	0.02
Google Web Light Disallow	6	0.01
Google BIG Bench Disallow	2	0