Robots.txt Usage Distribution in Rwanda

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Common Crawl Bot Disallow	225	7.19
Google AdsBot Disallow	213	6.81
Ahrefs Bot Disallow	199	6.36
Google AdsBot Mobile Disallow	197	6.3
Googlebot Disallow	183	5.85
Semrush Bot Disallow	180	5.75
Majestic Bot Disallow	179	5.72
Sogou Bot Disallow	174	5.56
Soso Bot Disallow	171	5.47
Exalead Bot Disallow	157	5.02
360 Bot Disallow	150	4.8
BuiltWithBot Disallow	150	4.8
Robots Disallow	97	3.1
GPTBot Disallow	83	2.65
AppleBot Disallow	79	2.53
Google AdsBot Mobile Apps Disallow	74	2.37
Anthropic Claude Bot Disallow	73	2.33
Google Bard Disallow	73	2.33
Cohere AI Disallow	73	2.33
Perplexity	73	2.33
Facebook Bot Disallow	72	2.3
Apple AI Bot Disallow	72	2.3
Yandex Bot Disallow	51	1.63
Microsoft BingBot Disallow	30	0.96
Yahoo Slurp Disallow	30	0.96
Alexa Bot Disallow	28	0.9
Google Media Partners Disallow	15 ^*	0.48
Microsoft MSNBot Disallow	5	0.16
Google Image Bot Disallow	5	0.16
DuckDuckGo Bot Disallow	3	0.1
Google News Bot Disallow	2	0.06
Google Video Bot Disallow	2	0.06
Google Favicon Disallow	2	0.06
YOU Disallow	2	0.06
Google API Bot Disallow	2	0.06
Google DuplexWeb Disallow	2	0.06
ByteDance Bytespider Disallow	1	0.03
Webz Omgili Disallow	1	0.03