Robots.txt Usage Distribution in Rwanda

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Sogou Bot Disallow	245	8.97
Googlebot Disallow	210	7.69
Common Crawl Bot Disallow	192	33.8
Ahrefs Bot Disallow	171	6.26
Soso Bot Disallow	164	6
Majestic Bot Disallow	156	5.71
Google AdsBot Disallow	155	5.67
Semrush Bot Disallow	153	5.6
Exalead Bot Disallow	144	5.27
360 Bot Disallow	142	5.2
BuiltWithBot Disallow	142	5.2
Google AdsBot Mobile Disallow	141	5.16
Robots Disallow	115	4.21
GPTBot Disallow	54	9.51
AppleBot Disallow	51	1.87
Anthropic Claude Bot Disallow	49	8.63
Perplexity	49	8.63
Google AdsBot Mobile Apps Disallow	49	1.79
Google Bard Disallow	48	8.45
Facebook Bot Disallow	48	8.45
Cohere AI Disallow	48	8.45
Apple AI Bot Disallow	48	8.45
Yandex Bot Disallow	35	1.28
Microsoft BingBot Disallow	30	1.1
Yahoo Slurp Disallow	27	0.99
Alexa Bot Disallow	25	4.4
Google Media Partners Disallow	13 ^*	0.48
Microsoft MSNBot Disallow	6	0.22
Google Image Bot Disallow	4	0.15
DuckDuckGo Bot Disallow	3	0.11
ByteDance Bytespider Disallow	2	0.35
Webz Omgili Disallow	2	0.35
Google News Bot Disallow	2	0.07
Google Video Bot Disallow	2	0.07
Google Favicon Disallow	2	0.07
Google API Bot Disallow	2	0.07
Google DuplexWeb Disallow	2	0.07
YOU Disallow	1	0.18