Robots.txt Usage Distribution in Philippines

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Some Robots Disallow	82,078	49.57
Googlebot Disallow	11,218	6.77
Baidu Bot Disallow	8,314	5.02
Sogou Bot Disallow	7,455	4.5
Google AdsBot Disallow	6,638	4.01
Yandex Bot Disallow	5,618	3.39
Alexa Bot Disallow	5,506	3.33
Microsoft BingBot Disallow	5,483	3.31
Yahoo Slurp Disallow	5,443	3.29
Soso Bot Disallow	5,386	3.25
Robots Disallow	5,358	3.24
Google AdsBot Mobile Disallow	4,025	2.43
Ahrefs Bot Disallow	2,298	1.39
GPTBot Disallow	917	0.55
Common Crawl Bot Disallow	901	0.54
Anthropic Claude Bot Disallow	865	0.52
Google Bard Disallow	863	0.52
AppleBot Disallow	862	0.52
Facebook Bot Disallow	835	0.5
Apple AI Bot Disallow	823	0.5
Google Media Partners Disallow	805 ^*	0.49
Perplexity	784	0.47
Cohere AI Disallow	770	0.46
Google AdsBot Mobile Apps Disallow	750	0.45
Semrush Bot Disallow	352	0.21
Majestic Bot Disallow	330	0.2
360 Bot Disallow	326	0.2
Microsoft MSNBot Disallow	99	0.06
Webz Omgili Disallow	95	0.06
ByteDance Bytespider Disallow	89	0.05
Exalead Bot Disallow	82	0.05
Google Image Bot Disallow	75	0.05
Diffbot Disallow	27	0.02
Google News Bot Disallow	17	0.01
BuiltWithBot Disallow	16	0.01
DuckDuckGo Bot Disallow	16	0.01
Google Video Bot Disallow	13	0.01
YOU Disallow	12	0.01
Google Favicon Disallow	10	0.01
Google API Bot Disallow	10	0.01
Youdao Bot Disallow	9	0.01
Pipl Disallow	9	0.01
Google DuplexWeb Disallow	8	0
Google FeedFetcher Disallow	3	0