Robots.txt Usage Distribution in Benin

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Robots Disallow	145	31.39
Googlebot Disallow	49	10.61
Baidu Bot Disallow	29	6.28
Ahrefs Bot Disallow	29	6.28
Yandex Bot Disallow	24	5.19
Sogou Bot Disallow	17	3.68
Majestic Bot Disallow	17	3.68
Semrush Bot Disallow	16	3.46
Google Media Partners Disallow	13 ^*	2.81
Microsoft BingBot Disallow	12	2.6
Alexa Bot Disallow	11	2.38
Yahoo Slurp Disallow	10	2.16
Soso Bot Disallow	9	1.95
GPTBot Disallow	9	1.95
Google Bard Disallow	7	1.52
Google AdsBot Mobile Apps Disallow	7	1.52
Common Crawl Bot Disallow	6	1.3
AppleBot Disallow	5	1.08
Anthropic Claude Bot Disallow	5	1.08
Facebook Bot Disallow	5	1.08
Cohere AI Disallow	5	1.08
Apple AI Bot Disallow	5	1.08
Perplexity	5	1.08
Google Image Bot Disallow	3	0.65
Exalead Bot Disallow	2	0.43
Microsoft MSNBot Disallow	2	0.43
Google News Bot Disallow	2	0.43
Google Video Bot Disallow	2	0.43
DuckDuckGo Bot Disallow	2	0.43
Google Favicon Disallow	2	0.43
Google API Bot Disallow	2	0.43
Google DuplexWeb Disallow	2	0.43
360 Bot Disallow	1	0.22
Youdao Bot Disallow	1	0.22
Pipl Disallow	1	0.22