Robots.txt Usage Distribution in the Top 10k Sites

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Robots Disallow	3,424	34.24
GPTBot Disallow	1,418	14.18
Some Robots Disallow	1,288	12.88
Googlebot Disallow	1,220	12.2
Common Crawl Bot Disallow	1,044	10.44
Ahrefs Bot Disallow	927	9.27
Anthropic Claude Bot Disallow	880	8.8
AppleBot Disallow	838	8.38
Google AdsBot Disallow	778	7.78
Yandex Bot Disallow	706	7.06
Google Bard Disallow	685	6.85
Semrush Bot Disallow	684	6.84
Facebook Bot Disallow	645	6.45
ByteDance Bytespider Disallow	639	6.39
Perplexity	615	6.15
Apple AI Bot Disallow	591	5.91
Webz Omgili Disallow	591	5.91
Cohere AI Disallow	541	5.41
Baidu Bot Disallow	533	5.33
Majestic Bot Disallow	526	5.26
Microsoft BingBot Disallow	514	5.14
Diffbot Disallow	399	3.99
Alexa Bot Disallow	390	3.9
Yahoo Slurp Disallow	301	3.01
Microsoft MSNBot Disallow	278	2.78
Google Media Partners Disallow	273	2.73
Google AdsBot Mobile Disallow	250	2.5
Sogou Bot Disallow	232	2.32
Google Image Bot Disallow	228	2.28
Exalead Bot Disallow	184	1.84
Google News Bot Disallow	169	1.69
Google AdsBot Mobile Apps Disallow	156	1.56
YOU Disallow	141	1.41
Soso Bot Disallow	139	1.39
DuckDuckGo Bot Disallow	123	1.23
360 Bot Disallow	115	1.15
Google Video Bot Disallow	94	0.94
Google API Bot Disallow	68	0.68
Google Favicon Disallow	63	0.63
Pipl Disallow	61	0.61
Google DuplexWeb Disallow	57	0.57
BuiltWithBot Disallow	43	0.43
Youdao Bot Disallow	23	0.23
Google FeedFetcher Disallow	2	0.02