Robots.txt Usage Distribution in Cocos Islands

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	38,799	12.64
Baidu Bot Disallow	36,770	11.98
Sogou Bot Disallow	33,766	11
Yandex Bot Disallow	27,096	8.83
Microsoft BingBot Disallow	26,277	8.56
Yahoo Slurp Disallow	26,272	8.56
Alexa Bot Disallow	25,629	77.33
Soso Bot Disallow	24,142	7.87
Some Robots Disallow	19,334	6.3
Robots Disallow	16,723	5.45
Ahrefs Bot Disallow	4,225	1.38
Google Media Partners Disallow	3,980 ^*	1.3
Majestic Bot Disallow	3,885	1.27
Semrush Bot Disallow	3,638	1.19
360 Bot Disallow	1,960	0.64
Microsoft MSNBot Disallow	1,873	0.61
Google AdsBot Disallow	1,382	0.45
Google Image Bot Disallow	1,139	0.37
GPTBot Disallow	1,017	3.07
Common Crawl Bot Disallow	939	2.83
Youdao Bot Disallow	785	0.26
Anthropic Claude Bot Disallow	767	2.31
Google AdsBot Mobile Disallow	747	0.24
AppleBot Disallow	736	0.24
ByteDance Bytespider Disallow	702	2.12
Facebook Bot Disallow	544	1.64
Google Bard Disallow	520	1.57
Exalead Bot Disallow	484	0.16
Apple AI Bot Disallow	394	1.19
Perplexity	350	1.06
Cohere AI Disallow	346	1.04
Google AdsBot Mobile Apps Disallow	326	0.11
Webz Omgili Disallow	220	0.66
BuiltWithBot Disallow	173	0.06
DuckDuckGo Bot Disallow	132	0.04
Google News Bot Disallow	124	0.04
YOU Disallow	119	0.36
Diffbot Disallow	107	0.32
Google Video Bot Disallow	107	0.03
Google Favicon Disallow	104	0.03
Google API Bot Disallow	96	0.03
Google DuplexWeb Disallow	95	0.03
Pipl Disallow	28	0.01
Google FeedFetcher Disallow	15	0
LLMS Text	2	0