Robots.txt Usage Distribution in Thailand

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	11,479	12.76
Google AdsBot Disallow	8,852	9.84
Google AdsBot Mobile Disallow	7,728	8.59
Alexa Bot Disallow	6,933	7.71
Ahrefs Bot Disallow	6,393	7.11
Majestic Bot Disallow	6,004	6.67
Semrush Bot Disallow	5,989	6.66
Some Robots Disallow	5,449	6.06
Baidu Bot Disallow	4,818	5.36
Yandex Bot Disallow	3,299	3.67
Robots Disallow	2,520	2.8
Sogou Bot Disallow	2,419	2.69
Microsoft BingBot Disallow	2,061	2.29
Common Crawl Bot Disallow	1,992	2.21
Yahoo Slurp Disallow	1,987	2.21
Soso Bot Disallow	1,656	1.84
Google Media Partners Disallow	1,186 ^*	1.32
Exalead Bot Disallow	1,157	1.29
GPTBot Disallow	885	0.98
Anthropic Claude Bot Disallow	831	0.92
AppleBot Disallow	823	0.91
Google Bard Disallow	773	0.86
Facebook Bot Disallow	760	0.84
Cohere AI Disallow	734	0.82
Perplexity	732	0.81
Apple AI Bot Disallow	723	0.8
Google AdsBot Mobile Apps Disallow	699	0.78
DuckDuckGo Bot Disallow	279	0.31
ByteDance Bytespider Disallow	146	0.16
360 Bot Disallow	128	0.14
Google Image Bot Disallow	109	0.12
Microsoft MSNBot Disallow	104	0.12
Webz Omgili Disallow	70	0.08
BuiltWithBot Disallow	58	0.06
Diffbot Disallow	41	0.05
YOU Disallow	30	0.03
Google News Bot Disallow	25	0.03
Google Video Bot Disallow	20	0.02
Youdao Bot Disallow	19	0.02
Google Favicon Disallow	16	0.02
Pipl Disallow	16	0.02
Google API Bot Disallow	12	0.01
Google DuplexWeb Disallow	12	0.01