Robots.txt Usage Distribution in Thailand

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	11,680	10.51
Ahrefs Bot Disallow	10,682	9.61
Majestic Bot Disallow	10,247	9.22
Alexa Bot Disallow	10,222	9.2
Semrush Bot Disallow	10,221	9.2
Google AdsBot Disallow	9,441	8.5
Google AdsBot Mobile Disallow	8,289	7.46
Some Robots Disallow	5,413	4.87
Baidu Bot Disallow	5,035	4.53
Yandex Bot Disallow	3,699	3.33
Common Crawl Bot Disallow	2,867	2.58
Robots Disallow	2,385	2.15
Microsoft BingBot Disallow	1,946	1.75
Yahoo Slurp Disallow	1,880	1.69
Sogou Bot Disallow	1,701	1.53
Exalead Bot Disallow	1,561	1.4
Soso Bot Disallow	1,378	1.24
GPTBot Disallow	1,248	1.12
AppleBot Disallow	1,209	1.09
Anthropic Claude Bot Disallow	1,190	1.07
Google Bard Disallow	1,103	0.99
Google Media Partners Disallow	1,087 ^*	0.98
Facebook Bot Disallow	1,084	0.98
Cohere AI Disallow	1,055	0.95
Perplexity	1,054	0.95
Apple AI Bot Disallow	1,052	0.95
Google AdsBot Mobile Apps Disallow	1,040	0.94
DuckDuckGo Bot Disallow	400	0.36
ByteDance Bytespider Disallow	154	0.14
360 Bot Disallow	154	0.14
Google Image Bot Disallow	142	0.13
Microsoft MSNBot Disallow	131	0.12
BuiltWithBot Disallow	72	0.06
Webz Omgili Disallow	69	0.06
Google News Bot Disallow	38	0.03
Diffbot Disallow	36	0.03
Google Video Bot Disallow	27	0.02
YOU Disallow	26	0.02
Youdao Bot Disallow	25	0.02
Google Favicon Disallow	21	0.02
Google API Bot Disallow	17	0.02
Google DuplexWeb Disallow	16	0.01
Pipl Disallow	16	0.01
Google FeedFetcher Disallow	1	0