Robots.txt Usage Distribution in Angola

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	183	15.46
Googlebot Disallow	178	15.03
Robots Disallow	175	14.78
Google AdsBot Mobile Disallow	159	13.43
Ahrefs Bot Disallow	42	3.55
Baidu Bot Disallow	32	2.7
Common Crawl Bot Disallow	28	11.81
GPTBot Disallow	27	11.39
Yandex Bot Disallow	25	2.11
Anthropic Claude Bot Disallow	25	10.55
Google Bard Disallow	25	10.55
AppleBot Disallow	24	2.03
Facebook Bot Disallow	24	10.13
Apple AI Bot Disallow	23	9.7
Perplexity	23	9.7
Cohere AI Disallow	22	9.28
Microsoft BingBot Disallow	21	1.77
Alexa Bot Disallow	21	8.86
Yahoo Slurp Disallow	20	1.69
Google AdsBot Mobile Apps Disallow	20	1.69
Soso Bot Disallow	17	1.44
Majestic Bot Disallow	16	1.35
Google Media Partners Disallow	13 ^*	1.1
Semrush Bot Disallow	12	1.01
ByteDance Bytespider Disallow	6	2.53
Exalead Bot Disallow	4	0.34
Webz Omgili Disallow	4	1.69
Microsoft MSNBot Disallow	3	0.25
Google Image Bot Disallow	2	0.17
Diffbot Disallow	2	0.84
360 Bot Disallow	1	0.08
YOU Disallow	1	0.42
Google News Bot Disallow	1	0.08
Google Video Bot Disallow	1	0.08
DuckDuckGo Bot Disallow	1	0.08
Google Favicon Disallow	1	0.08
Google API Bot Disallow	1	0.08
Google DuplexWeb Disallow	1	0.08