Robots.txt Usage Distribution in Taiwan

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	9,198	11.69
Baidu Bot Disallow	6,643	8.44
Google AdsBot Disallow	5,458	6.93
Sogou Bot Disallow	5,302	6.74
Robots Disallow	5,296	6.73
Ahrefs Bot Disallow	4,077	5.18
Majestic Bot Disallow	4,009	5.09
Yandex Bot Disallow	3,265	4.15
Anthropic Claude Bot Disallow	2,925	16.92
Google AdsBot Mobile Disallow	2,750	3.49
Semrush Bot Disallow	2,617	3.32
Some Robots Disallow	2,344	2.98
GPTBot Disallow	2,229	12.89
Microsoft BingBot Disallow	2,129	2.7
Common Crawl Bot Disallow	2,106	12.18
Yahoo Slurp Disallow	1,997	2.54
Alexa Bot Disallow	1,878	10.86
Soso Bot Disallow	1,829	2.32
AppleBot Disallow	1,401	1.78
Google Bard Disallow	1,195	6.91
Apple AI Bot Disallow	1,172	6.78
Facebook Bot Disallow	1,162	6.72
Cohere AI Disallow	1,081	6.25
Google Media Partners Disallow	1,022 ^*	1.3
ByteDance Bytespider Disallow	1,016	5.88
Google Image Bot Disallow	942	1.2
Webz Omgili Disallow	706	4.08
Perplexity	571	3.3
Diffbot Disallow	557	3.22
Google AdsBot Mobile Apps Disallow	539	0.68
Microsoft MSNBot Disallow	298	0.38
360 Bot Disallow	238	0.3
Exalead Bot Disallow	184	0.23
LLMS Text	110	0.14
DuckDuckGo Bot Disallow	80	0.1
Youdao Bot Disallow	68	0.09
Google Video Bot Disallow	59	0.07
Google News Bot Disallow	50	0.06
Google API Bot Disallow	47	0.06
YOU Disallow	45	0.26
Google Favicon Disallow	40	0.05
Google DuplexWeb Disallow	33	0.04
BuiltWithBot Disallow	31	0.04
Pipl Disallow	5	0.01
Google FeedFetcher Disallow	2	0
Google Read Aloud Disallow	1	0