Robots.txt Usage Distribution in Nigeria

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	2,230	13.11
Robots Disallow	1,511	8.88
Baidu Bot Disallow	1,352	7.95
Google AdsBot Disallow	1,194	7.02
Yandex Bot Disallow	954	5.61
Sogou Bot Disallow	879	5.17
Google AdsBot Mobile Disallow	877	5.16
Yahoo Slurp Disallow	850	5
Microsoft BingBot Disallow	847	4.98
Alexa Bot Disallow	821	4.83
Soso Bot Disallow	782	4.6
Some Robots Disallow	695	4.09
Ahrefs Bot Disallow	548	3.22
GPTBot Disallow	318	1.87
Google Media Partners Disallow	313 ^*	1.84
Semrush Bot Disallow	281	1.65
Majestic Bot Disallow	262	1.54
Common Crawl Bot Disallow	250	1.47
AppleBot Disallow	250	1.47
Google Bard Disallow	227	1.33
Anthropic Claude Bot Disallow	221	1.3
Facebook Bot Disallow	207	1.22
Apple AI Bot Disallow	200	1.18
Perplexity	197	1.16
Cohere AI Disallow	193	1.13
Google AdsBot Mobile Apps Disallow	190	1.12
Microsoft MSNBot Disallow	65	0.38
Exalead Bot Disallow	61	0.36
ByteDance Bytespider Disallow	60	0.35
Google Image Bot Disallow	39	0.23
Webz Omgili Disallow	28	0.16
360 Bot Disallow	25	0.15
Youdao Bot Disallow	12	0.07
BuiltWithBot Disallow	11	0.06
DuckDuckGo Bot Disallow	10	0.06
YOU Disallow	10	0.06
Diffbot Disallow	7	0.04
Google News Bot Disallow	7	0.04
Google API Bot Disallow	6	0.04
Google Video Bot Disallow	4	0.02
Google Favicon Disallow	4	0.02
Google DuplexWeb Disallow	4	0.02
Pipl Disallow	3	0.02
Google FeedFetcher Disallow	1	0.01
Google Web Light Disallow	1	0.01