Robots.txt Usage Distribution in Thailand

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	11,385	9.97
Alexa Bot Disallow	11,260	9.87
Ahrefs Bot Disallow	10,892	9.54
Majestic Bot Disallow	10,402	9.11
Semrush Bot Disallow	10,373	9.09
Google AdsBot Disallow	8,888	7.79
Google AdsBot Mobile Disallow	7,782	6.82
Some Robots Disallow	5,387	4.72
Baidu Bot Disallow	4,751	4.16
Yandex Bot Disallow	4,154	3.64
Common Crawl Bot Disallow	2,898	2.54
Robots Disallow	2,782	2.44
Microsoft BingBot Disallow	2,383	2.09
Yahoo Slurp Disallow	2,304	2.02
Sogou Bot Disallow	2,067	1.81
Soso Bot Disallow	1,788	1.57
Exalead Bot Disallow	1,575	1.38
Google Media Partners Disallow	1,403 ^*	1.23
GPTBot Disallow	1,273	1.12
AppleBot Disallow	1,243	1.09
Anthropic Claude Bot Disallow	1,219	1.07
Google Bard Disallow	1,129	0.99
Facebook Bot Disallow	1,112	0.97
Perplexity	1,080	0.95
Cohere AI Disallow	1,079	0.95
Apple AI Bot Disallow	1,072	0.94
Google AdsBot Mobile Apps Disallow	1,051	0.92
DuckDuckGo Bot Disallow	406	0.36
ByteDance Bytespider Disallow	171	0.15
360 Bot Disallow	155	0.14
Google Image Bot Disallow	146	0.13
Microsoft MSNBot Disallow	134	0.12
Webz Omgili Disallow	79	0.07
BuiltWithBot Disallow	71	0.06
Diffbot Disallow	47	0.04
Google News Bot Disallow	37	0.03
YOU Disallow	32	0.03
Google Video Bot Disallow	27	0.02
Youdao Bot Disallow	27	0.02
Google Favicon Disallow	21	0.02
Pipl Disallow	19	0.02
Google API Bot Disallow	17	0.01
Google DuplexWeb Disallow	16	0.01
Google FeedFetcher Disallow	1	0