Robots.txt Usage Distribution in Burkina Faso

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Robots Disallow	173	31.98
Googlebot Disallow	42	7.76
Google AdsBot Disallow	41	7.58
Ahrefs Bot Disallow	29	5.36
Baidu Bot Disallow	28	5.18
Google AdsBot Mobile Disallow	28	5.18
Majestic Bot Disallow	19	3.51
Semrush Bot Disallow	18	3.33
Yandex Bot Disallow	16	2.96
Google Media Partners Disallow	13 ^*	2.4
GPTBot Disallow	12	15
Common Crawl Bot Disallow	11	13.75
Microsoft BingBot Disallow	8	1.48
Alexa Bot Disallow	8	10
Anthropic Claude Bot Disallow	8	10
Yahoo Slurp Disallow	7	1.29
AppleBot Disallow	7	1.29
Facebook Bot Disallow	7	8.75
Cohere AI Disallow	7	8.75
Apple AI Bot Disallow	7	8.75
Perplexity	7	8.75
Google AdsBot Mobile Apps Disallow	7	1.29
Soso Bot Disallow	5	0.92
Google Bard Disallow	5	6.25
Microsoft MSNBot Disallow	5	0.92
Webz Omgili Disallow	3	3.75
Google Image Bot Disallow	3	0.55
ByteDance Bytespider Disallow	2	2.5
Diffbot Disallow	2	2.5
Google News Bot Disallow	2	0.37
Google Video Bot Disallow	2	0.37
DuckDuckGo Bot Disallow	2	0.37
Google Favicon Disallow	2	0.37
Google API Bot Disallow	2	0.37
Google DuplexWeb Disallow	2	0.37
Exalead Bot Disallow	1	0.18