Robots.txt Usage Distribution in Seychelles

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	265	13.85
Sogou Bot Disallow	168	8.78
Baidu Bot Disallow	164	8.57
Google Media Partners Disallow	133 ^*	6.95
Robots Disallow	124	6.48
Soso Bot Disallow	124	6.48
Alexa Bot Disallow	119	39.14
Yandex Bot Disallow	116	6.06
Microsoft BingBot Disallow	114	5.96
Yahoo Slurp Disallow	114	5.96
Some Robots Disallow	92	4.81
Ahrefs Bot Disallow	44	2.3
Common Crawl Bot Disallow	33	10.86
Majestic Bot Disallow	31	1.62
Semrush Bot Disallow	28	1.46
GPTBot Disallow	25	8.22
Anthropic Claude Bot Disallow	23	7.57
AppleBot Disallow	21	1.1
Google Bard Disallow	19	6.25
Facebook Bot Disallow	19	6.25
Apple AI Bot Disallow	18	5.92
Exalead Bot Disallow	17	0.89
Cohere AI Disallow	16	5.26
Perplexity	16	5.26
Google AdsBot Mobile Apps Disallow	15	0.78
360 Bot Disallow	15	0.78
BuiltWithBot Disallow	15	0.78
Webz Omgili Disallow	4	1.32
ByteDance Bytespider Disallow	3	0.99
Microsoft MSNBot Disallow	3	0.16
Google Video Bot Disallow	3	0.16
Google Image Bot Disallow	2	0.1
DuckDuckGo Bot Disallow	2	0.1
Diffbot Disallow	1	0.33
YOU Disallow	1	0.33
Google News Bot Disallow	1	0.05
Google Favicon Disallow	1	0.05
Youdao Bot Disallow	1	0.05
Google API Bot Disallow	1	0.05
Google DuplexWeb Disallow	1	0.05
LLMS Text	1	0.05