Robots.txt Usage Distribution in Kenya

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Robots Disallow	2,266	12.45
Googlebot Disallow	1,873	10.29
Google AdsBot Disallow	1,658	9.11
Google AdsBot Mobile Disallow	1,280	7.03
Baidu Bot Disallow	831	4.57
Google Media Partners Disallow	744 ^*	4.09
Microsoft BingBot Disallow	670	3.68
Yandex Bot Disallow	640	3.52
Sogou Bot Disallow	584	3.21
Alexa Bot Disallow	570	3.13
Yahoo Slurp Disallow	559	3.07
Soso Bot Disallow	540	2.97
Some Robots Disallow	525	2.89
Ahrefs Bot Disallow	524	2.88
GPTBot Disallow	513	2.82
Common Crawl Bot Disallow	469	2.58
Google Bard Disallow	438	2.41
Anthropic Claude Bot Disallow	437	2.4
AppleBot Disallow	425	2.34
Facebook Bot Disallow	420	2.31
Apple AI Bot Disallow	415	2.28
Perplexity	411	2.26
Cohere AI Disallow	409	2.25
Google AdsBot Mobile Apps Disallow	408	2.24
Majestic Bot Disallow	143	0.79
Semrush Bot Disallow	132	0.73
ByteDance Bytespider Disallow	42	0.23
Microsoft MSNBot Disallow	40	0.22
Google Image Bot Disallow	40	0.22
Exalead Bot Disallow	34	0.19
Webz Omgili Disallow	26	0.14
360 Bot Disallow	21	0.12
Google News Bot Disallow	15	0.08
Google Video Bot Disallow	12	0.07
DuckDuckGo Bot Disallow	12	0.07
Google Favicon Disallow	12	0.07
Google API Bot Disallow	11	0.06
Google DuplexWeb Disallow	11	0.06
Diffbot Disallow	10	0.05
BuiltWithBot Disallow	9	0.05
YOU Disallow	7	0.04
Youdao Bot Disallow	6	0.03
Pipl Disallow	2	0.01
Google FeedFetcher Disallow	1	0.01