Robots.txt Usage Distribution in East Timor

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	44	13.33
Robots Disallow	43	13.03
Some Robots Disallow	24	7.27
Yandex Bot Disallow	20	6.06
Microsoft BingBot Disallow	19	5.76
Yahoo Slurp Disallow	17	5.15
Alexa Bot Disallow	16	4.85
Soso Bot Disallow	14	4.24
Common Crawl Bot Disallow	11	3.33
AppleBot Disallow	10	3.03
Ahrefs Bot Disallow	9	2.73
GPTBot Disallow	9	2.73
Anthropic Claude Bot Disallow	9	2.73
Google Bard Disallow	9	2.73
Facebook Bot Disallow	9	2.73
Google AdsBot Mobile Apps Disallow	8	2.42
Apple AI Bot Disallow	7	2.12
Cohere AI Disallow	7	2.12
Perplexity	7	2.12
Majestic Bot Disallow	5	1.52
Semrush Bot Disallow	5	1.52
ByteDance Bytespider Disallow	3	0.91
Microsoft MSNBot Disallow	3	0.91
Webz Omgili Disallow	3	0.91
Google Image Bot Disallow	3	0.91
Exalead Bot Disallow	2	0.61
Google News Bot Disallow	2	0.61
Google Video Bot Disallow	2	0.61
DuckDuckGo Bot Disallow	2	0.61
Google Favicon Disallow	2	0.61
Google API Bot Disallow	2	0.61
Google DuplexWeb Disallow	2	0.61
360 Bot Disallow	1	0.3
BuiltWithBot Disallow	1	0.3