Robots.txt Usage Distribution in Sao Tome and Principe

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	276	11.12
Baidu Bot Disallow	250	10.07
Robots Disallow	214	8.62
Sogou Bot Disallow	164	6.61
Yandex Bot Disallow	161	6.49
Microsoft BingBot Disallow	159	6.41
Alexa Bot Disallow	157	6.33
Yahoo Slurp Disallow	156	6.29
Soso Bot Disallow	151	6.08
Google Media Partners Disallow	134 ^*	5.4
Some Robots Disallow	113	4.55
GPTBot Disallow	50	2.01
Common Crawl Bot Disallow	43	1.73
Google Bard Disallow	42	1.69
Google AdsBot Disallow	40	1.61
Anthropic Claude Bot Disallow	39	1.57
AppleBot Disallow	36	1.45
Facebook Bot Disallow	36	1.45
Ahrefs Bot Disallow	32	1.29
Webz Omgili Disallow	32	1.29
Apple AI Bot Disallow	29	1.17
Majestic Bot Disallow	22	0.89
Semrush Bot Disallow	21	0.85
Cohere AI Disallow	17	0.68
Perplexity	16	0.64
ByteDance Bytespider Disallow	13	0.52
Google AdsBot Mobile Apps Disallow	11	0.44
Microsoft MSNBot Disallow	9	0.36
Exalead Bot Disallow	8	0.32
Diffbot Disallow	7	0.28
YOU Disallow	7	0.28
Google Image Bot Disallow	6	0.24
360 Bot Disallow	4	0.16
Google News Bot Disallow	4	0.16
Google Video Bot Disallow	4	0.16
DuckDuckGo Bot Disallow	4	0.16
BuiltWithBot Disallow	3	0.12
Google Favicon Disallow	3	0.12
Google API Bot Disallow	3	0.12
Google DuplexWeb Disallow	3	0.12
Pipl Disallow	2	0.08
Youdao Bot Disallow	1	0.04