Robots.txt Usage Distribution in Mauritania

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Robots Disallow	25	17.61
Yandex Bot Disallow	13	9.15
Sogou Bot Disallow	11	7.75
Ahrefs Bot Disallow	9	6.34
Alexa Bot Disallow	9	6.34
Microsoft BingBot Disallow	8	5.63
Yahoo Slurp Disallow	8	5.63
Soso Bot Disallow	7	4.93
Google AdsBot Mobile Apps Disallow	5	3.52
Majestic Bot Disallow	5	3.52
GPTBot Disallow	4	2.82
AppleBot Disallow	4	2.82
Perplexity	4	2.82
Semrush Bot Disallow	4	2.82
Common Crawl Bot Disallow	3	2.11
Anthropic Claude Bot Disallow	3	2.11
Google Bard Disallow	3	2.11
Facebook Bot Disallow	3	2.11
Apple AI Bot Disallow	3	2.11
Cohere AI Disallow	3	2.11
Microsoft MSNBot Disallow	1	0.7
Google Image Bot Disallow	1	0.7
Google News Bot Disallow	1	0.7
Google Video Bot Disallow	1	0.7
DuckDuckGo Bot Disallow	1	0.7
Google Favicon Disallow	1	0.7
Google API Bot Disallow	1	0.7
Google DuplexWeb Disallow	1	0.7