Robots.txt Usage Distribution on the Entire Internet

Distribution for websites using Robots.txt technologies

Top In Robots.txt Usage Distribution on the Entire Internet

Technology	Websites	%
Some Robots Disallow	39,138,226	23.93
Googlebot Disallow	13,834,398	8.46
Baidu Bot Disallow	10,188,318	6.23
Sogou Bot Disallow	9,557,148	5.84
Google Media Partners Disallow	9,287,522	5.68
Google AdsBot Disallow	8,717,405	5.33
Google AdsBot Mobile Disallow	7,269,025	4.45
Robots Disallow	5,581,000	3.41
Yandex Bot Disallow	5,356,402	3.28
Microsoft BingBot Disallow	5,195,571	3.18
Alexa Bot Disallow	4,883,371	2.99
Yahoo Slurp Disallow	4,749,920	2.9
Soso Bot Disallow	4,645,220	2.84
Ahrefs Bot Disallow	3,935,542	2.41
GPTBot Disallow	2,727,265	1.67
Common Crawl Bot Disallow	2,682,292	1.64
AppleBot Disallow	2,638,484	1.61
Anthropic Claude Bot Disallow	2,548,782	1.56
Google Bard Disallow	2,421,613	1.48
Facebook Bot Disallow	2,391,535	1.46
Cohere AI Disallow	2,305,690	1.41
Apple AI Bot Disallow	2,273,964	1.39
Perplexity	2,230,267	1.36
Google AdsBot Mobile Apps Disallow	2,114,262	1.29
Majestic Bot Disallow	1,938,772	1.19
Semrush Bot Disallow	1,711,598	1.05
ByteDance Bytespider Disallow	641,870	0.39
Exalead Bot Disallow	395,923	0.24
Microsoft MSNBot Disallow	351,362	0.21
Webz Omgili Disallow	347,921	0.21
360 Bot Disallow	288,753	0.18
Google Image Bot Disallow	264,115	0.16
Diffbot Disallow	179,949	0.11
BuiltWithBot Disallow	156,116	0.1
YOU Disallow	105,454	0.06
Google News Bot Disallow	78,385	0.05
Google Video Bot Disallow	75,967	0.05
DuckDuckGo Bot Disallow	75,846	0.05
Google Favicon Disallow	65,330	0.04
Youdao Bot Disallow	64,292	0.04
Google API Bot Disallow	45,937	0.03
Google DuplexWeb Disallow	43,852	0.03
Pipl Disallow	15,760	0.01
Google FeedFetcher Disallow	3,223	0
Google Read Aloud Disallow	2,071	0
LLMS Text	1,288	0
Google Web Light Disallow	509	0
Google BIG Bench Disallow	24	0