Robots.txt Usage Distribution in United States

Distribution for websites using Robots.txt technologies

Top In Robots.txt Usage Distribution in United States

Technology	Websites	%
Sogou Bot Disallow	2,920,724	5.77
GPTBot Disallow	2,477,902	4.9
Common Crawl Bot Disallow	2,476,646	4.9
Anthropic Claude Bot Disallow	2,318,839	4.58
AppleBot Disallow	2,309,669	4.57
Ahrefs Bot Disallow	2,303,423	4.55
Google Bard Disallow	2,277,195	4.5
Facebook Bot Disallow	2,256,923	4.46
Google AdsBot Disallow	2,228,653	4.41
Apple AI Bot Disallow	2,222,825	4.39
Yandex Bot Disallow	2,206,766	4.36
Perplexity	2,132,486	4.22
Cohere AI Disallow	2,126,848	4.2
Google AdsBot Mobile Apps Disallow	2,073,142	4.1
Alexa Bot Disallow	2,049,550	4.05
Microsoft BingBot Disallow	2,045,124	4.04
Robots Disallow	2,018,445	3.99
Yahoo Slurp Disallow	1,938,071	3.83
Soso Bot Disallow	1,826,660	3.61
Google AdsBot Mobile Disallow	1,793,020	3.54
Googlebot Disallow	1,468,914	2.9
Majestic Bot Disallow	1,023,121	2.02
Semrush Bot Disallow	896,167	1.77
Baidu Bot Disallow	622,157	1.23
Some Robots Disallow	530,017	1.05
ByteDance Bytespider Disallow	307,862	0.61
Microsoft MSNBot Disallow	240,143	0.47
Exalead Bot Disallow	231,680	0.46
Google Media Partners Disallow	229,646 ^*	0.45
Webz Omgili Disallow	213,741	0.42
360 Bot Disallow	208,423	0.41
Google Image Bot Disallow	189,336	0.37
BuiltWithBot Disallow	79,953	0.16
Diffbot Disallow	77,888	0.15
Google News Bot Disallow	46,763	0.09
Google Video Bot Disallow	46,642	0.09
Youdao Bot Disallow	45,525	0.09
Google Favicon Disallow	38,138	0.08
DuckDuckGo Bot Disallow	33,019	0.07
YOU Disallow	21,575	0.04
Google API Bot Disallow	11,157	0.02
Pipl Disallow	11,088	0.02
Google DuplexWeb Disallow	10,703	0.02
Google FeedFetcher Disallow	1,932	0
Google Read Aloud Disallow	210	0
Google Web Light Disallow	48	0
Google BIG Bench Disallow	19	0