Robots.txt Usage Distribution in Mali

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google Media Partners Disallow	342 ^*	31.84
Sogou Bot Disallow	65	6.05
Yandex Bot Disallow	63	5.87
Alexa Bot Disallow	60	5.59
Yahoo Slurp Disallow	60	5.59
Soso Bot Disallow	60	5.59
Googlebot Disallow	56	5.21
Microsoft BingBot Disallow	56	5.21
Robots Disallow	54	5.03
Google AdsBot Disallow	52	4.84
Baidu Bot Disallow	17	1.58
GPTBot Disallow	17	1.58
Common Crawl Bot Disallow	15	1.4
Anthropic Claude Bot Disallow	14	1.3
Facebook Bot Disallow	14	1.3
Cohere AI Disallow	14	1.3
Perplexity	14	1.3
AppleBot Disallow	13	1.21
Apple AI Bot Disallow	13	1.21
Google Bard Disallow	12	1.12
Google AdsBot Mobile Apps Disallow	11	1.02
Some Robots Disallow	10	0.93
Ahrefs Bot Disallow	10	0.93
Semrush Bot Disallow	7	0.65
Majestic Bot Disallow	6	0.56
ByteDance Bytespider Disallow	4	0.37
Webz Omgili Disallow	3	0.28
Diffbot Disallow	3	0.28
Exalead Bot Disallow	2	0.19
Microsoft MSNBot Disallow	2	0.19
Google Image Bot Disallow	1	0.09
360 Bot Disallow	1	0.09
BuiltWithBot Disallow	1	0.09
Youdao Bot Disallow	1	0.09
YOU Disallow	1	0.09