Robots.txt Usage Distribution on the Entire Internet

Distribution for websites using Robots.txt technologies

Top In Robots.txt Usage Distribution on the Entire Internet

Technology	Websites	%
Some Robots Disallow	39,138,226	21.89
Googlebot Disallow	17,383,587	9.72
Baidu Bot Disallow	11,895,694	6.65
Google AdsBot Disallow	11,312,715	6.33
Google Media Partners Disallow	10,419,511	5.83
Google AdsBot Mobile Disallow	8,418,498	4.71
Sogou Bot Disallow	6,033,185	3.37
Yandex Bot Disallow	5,250,396	2.94
Robots Disallow	5,214,267	2.92
Microsoft BingBot Disallow	4,938,400	2.76
Ahrefs Bot Disallow	4,811,426	2.69
Alexa Bot Disallow	4,658,089	2.61
Yahoo Slurp Disallow	4,472,894	2.5
Soso Bot Disallow	4,272,238	2.39
GPTBot Disallow	3,576,292	2
Common Crawl Bot Disallow	3,539,018	1.98
AppleBot Disallow	3,469,718	1.94
Anthropic Claude Bot Disallow	3,318,915	1.86
Google Bard Disallow	3,195,201	1.79
Facebook Bot Disallow	3,159,803	1.77
Apple AI Bot Disallow	3,041,356	1.7
Cohere AI Disallow	2,965,441	1.66
Perplexity	2,942,952	1.65
Google AdsBot Mobile Apps Disallow	2,857,914	1.6
Majestic Bot Disallow	2,488,954	1.39
Semrush Bot Disallow	2,211,971	1.24
ByteDance Bytespider Disallow	769,409	0.43
Exalead Bot Disallow	510,010	0.29
Microsoft MSNBot Disallow	474,162	0.27
Webz Omgili Disallow	398,731	0.22
Google Image Bot Disallow	366,005	0.2
360 Bot Disallow	338,425	0.19
BuiltWithBot Disallow	163,027	0.09
Diffbot Disallow	135,688	0.08
Google News Bot Disallow	102,998	0.06
Google Video Bot Disallow	101,064	0.06
DuckDuckGo Bot Disallow	94,271	0.05
Google Favicon Disallow	84,983	0.05
Youdao Bot Disallow	72,532	0.04
YOU Disallow	60,856	0.03
Google API Bot Disallow	54,061	0.03
Google DuplexWeb Disallow	52,957	0.03
Pipl Disallow	20,657	0.01
Google FeedFetcher Disallow	3,805	0
Google Read Aloud Disallow	418	0
Google Web Light Disallow	106	0
Google BIG Bench Disallow	31	0