Robots.txt Usage Distribution in the Top 100k Sites

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Robots Disallow	30,148	30.15
Googlebot Disallow	9,749	9.75
Some Robots Disallow	9,200	9.2
Ahrefs Bot Disallow	9,023	9.02
GPTBot Disallow	8,497	8.5
Yandex Bot Disallow	6,543	6.54
Semrush Bot Disallow	6,265	6.26
AppleBot Disallow	6,252	6.25
Common Crawl Bot Disallow	6,107	6.11
Google AdsBot Disallow	5,685	5.68
Majestic Bot Disallow	5,449	5.45
Anthropic Claude Bot Disallow	5,361	5.36
Baidu Bot Disallow	4,809	4.81
Microsoft BingBot Disallow	4,457	4.46
Google Bard Disallow	4,392	4.39
ByteDance Bytespider Disallow	3,857	3.86
Facebook Bot Disallow	3,605	3.6
Perplexity	3,315	3.32
Webz Omgili Disallow	3,103	3.1
Alexa Bot Disallow	3,007	3.01
Cohere AI Disallow	2,956	2.96
Apple AI Bot Disallow	2,937	2.94
Yahoo Slurp Disallow	2,712	2.71
Google Media Partners Disallow	2,632	2.63
Microsoft MSNBot Disallow	2,446	2.45
Sogou Bot Disallow	2,256	2.26
Google Image Bot Disallow	2,164	2.16
Google AdsBot Mobile Disallow	2,128	2.13
Diffbot Disallow	1,929	1.93
Exalead Bot Disallow	1,716	1.72
Google News Bot Disallow	1,500	1.5
Soso Bot Disallow	1,401	1.4
Google AdsBot Mobile Apps Disallow	1,364	1.36
YOU Disallow	1,246	1.25
DuckDuckGo Bot Disallow	1,205	1.2
360 Bot Disallow	1,062	1.06
Google Video Bot Disallow	1,018	1.02
Google Favicon Disallow	819	0.82
Google API Bot Disallow	819	0.82
Google DuplexWeb Disallow	777	0.78
BuiltWithBot Disallow	571	0.57
Pipl Disallow	518	0.52
Youdao Bot Disallow	205	0.2
LLMS Text	87	0.09
Google FeedFetcher Disallow	31	0.03
Google Read Aloud Disallow	18	0.02
Google Web Light Disallow	8	0.01
Google BIG Bench Disallow	3	0