Robots.txt Usage Distribution in Madagascar

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Robots Disallow	206	20.68
Googlebot Disallow	117	11.75
Google AdsBot Disallow	88	8.84
Google AdsBot Mobile Disallow	81	8.13
Baidu Bot Disallow	33	3.31
GPTBot Disallow	28	12.28
Common Crawl Bot Disallow	26	11.4
Yandex Bot Disallow	25	2.51
Microsoft BingBot Disallow	25	2.51
Yahoo Slurp Disallow	24	2.41
Some Robots Disallow	23	2.31
AppleBot Disallow	23	2.31
Anthropic Claude Bot Disallow	23	10.09
Google Bard Disallow	23	10.09
Facebook Bot Disallow	23	10.09
Apple AI Bot Disallow	23	10.09
Perplexity	23	10.09
Cohere AI Disallow	22	9.65
Ahrefs Bot Disallow	20	2.01
Google AdsBot Mobile Apps Disallow	20	2.01
Alexa Bot Disallow	18	7.89
Soso Bot Disallow	18	1.81
Majestic Bot Disallow	14	1.41
Semrush Bot Disallow	14	1.41
Google Media Partners Disallow	13 ^*	1.31
ByteDance Bytespider Disallow	6	2.63
DuckDuckGo Bot Disallow	5	0.5
Exalead Bot Disallow	4	0.4
Webz Omgili Disallow	4	1.75
Google Image Bot Disallow	4	0.4
Microsoft MSNBot Disallow	3	0.3
Diffbot Disallow	3	1.32
YOU Disallow	3	1.32
360 Bot Disallow	2	0.2
BuiltWithBot Disallow	2	0.2
Google News Bot Disallow	2	0.2
Google Video Bot Disallow	1	0.1
Google Favicon Disallow	1	0.1
Google API Bot Disallow	1	0.1
Google DuplexWeb Disallow	1	0.1
LLMS Text	1	0.1