Robots.txt Usage Distribution in Philippines

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Some Robots Disallow	49,761	40.75
Googlebot Disallow	8,518	6.97
Baidu Bot Disallow	6,516	5.34
Sogou Bot Disallow	6,303	5.16
Yandex Bot Disallow	5,367	4.39
Robots Disallow	5,282	4.33
Alexa Bot Disallow	5,256	43.91
Microsoft BingBot Disallow	5,242	4.29
Yahoo Slurp Disallow	5,207	4.26
Soso Bot Disallow	5,172	4.24
Google AdsBot Disallow	4,655	3.81
Google AdsBot Mobile Disallow	2,810	2.3
Ahrefs Bot Disallow	1,985	1.63
GPTBot Disallow	860	7.18
Common Crawl Bot Disallow	846	7.07
AppleBot Disallow	821	0.67
Anthropic Claude Bot Disallow	816	6.82
Google Bard Disallow	810	6.77
Facebook Bot Disallow	790	6.6
Apple AI Bot Disallow	779	6.51
Google Media Partners Disallow	756 ^*	0.62
Perplexity	740	6.18
Cohere AI Disallow	730	6.1
Google AdsBot Mobile Apps Disallow	710	0.58
Semrush Bot Disallow	298	0.24
Majestic Bot Disallow	290	0.24
360 Bot Disallow	276	0.23
Webz Omgili Disallow	91	0.76
ByteDance Bytespider Disallow	80	0.67
Microsoft MSNBot Disallow	78	0.06
Google Image Bot Disallow	69	0.06
Exalead Bot Disallow	64	0.05
Diffbot Disallow	27	0.23
BuiltWithBot Disallow	16	0.01
Google News Bot Disallow	16	0.01
YOU Disallow	14	0.12
DuckDuckGo Bot Disallow	14	0.01
Google Video Bot Disallow	12	0.01
Google Favicon Disallow	9	0.01
Youdao Bot Disallow	9	0.01
Google API Bot Disallow	9	0.01
Google DuplexWeb Disallow	8	0.01
Pipl Disallow	8	0.01
Google FeedFetcher Disallow	1	0
LLMS Text	1	0