Robots.txt Usage Distribution in Mozambique

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	254	15.69
Google AdsBot Mobile Disallow	233	14.39
Googlebot Disallow	227	14.02
Robots Disallow	182	11.24
Google Media Partners Disallow	74 ^*	4.57
Common Crawl Bot Disallow	50	3.09
GPTBot Disallow	48	2.96
Ahrefs Bot Disallow	46	2.84
AppleBot Disallow	44	2.72
Anthropic Claude Bot Disallow	44	2.72
Google AdsBot Mobile Apps Disallow	44	2.72
Google Bard Disallow	43	2.66
Facebook Bot Disallow	43	2.66
Apple AI Bot Disallow	43	2.66
Cohere AI Disallow	43	2.66
Perplexity	43	2.66
Majestic Bot Disallow	28	1.73
Yandex Bot Disallow	25	1.54
Semrush Bot Disallow	25	1.54
Microsoft BingBot Disallow	13	0.8
Sogou Bot Disallow	12	0.74
Alexa Bot Disallow	12	0.74
Yahoo Slurp Disallow	10	0.62
Soso Bot Disallow	10	0.62
Exalead Bot Disallow	6	0.37
ByteDance Bytespider Disallow	4	0.25
Microsoft MSNBot Disallow	4	0.25
Google Image Bot Disallow	3	0.19
360 Bot Disallow	3	0.19
BuiltWithBot Disallow	2	0.12
Webz Omgili Disallow	1	0.06