Robots.txt Usage Distribution in South Korea

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	39,873	15.97
Google AdsBot Disallow	23,251	9.31
Google AdsBot Mobile Disallow	22,024	8.82
Some Robots Disallow	20,452	8.19
Robots Disallow	18,688	7.48
Baidu Bot Disallow	10,841	4.34
Microsoft BingBot Disallow	9,736	3.9
Yandex Bot Disallow	8,371	3.35
Sogou Bot Disallow	7,196	2.88
Yahoo Slurp Disallow	7,104	2.85
Alexa Bot Disallow	6,802	2.72
Soso Bot Disallow	6,275	2.51
Ahrefs Bot Disallow	6,155	2.47
Majestic Bot Disallow	5,773	2.31
Common Crawl Bot Disallow	5,660	2.27
GPTBot Disallow	5,315	2.13
Semrush Bot Disallow	5,281	2.12
Anthropic Claude Bot Disallow	4,633	1.86
Facebook Bot Disallow	4,185	1.68
Google Bard Disallow	4,137	1.66
Cohere AI Disallow	3,886	1.56
ByteDance Bytespider Disallow	3,803	1.52
Webz Omgili Disallow	3,168	1.27
Diffbot Disallow	2,917	1.17
AppleBot Disallow	2,448	0.98
Microsoft MSNBot Disallow	2,189	0.88
Google Image Bot Disallow	1,761	0.71
Apple AI Bot Disallow	1,310	0.52
Google Media Partners Disallow	1,146 ^*	0.46
Perplexity	1,133	0.45
Google AdsBot Mobile Apps Disallow	1,009	0.4
Exalead Bot Disallow	939	0.38
360 Bot Disallow	726	0.29
DuckDuckGo Bot Disallow	632	0.25
Pipl Disallow	272	0.11
BuiltWithBot Disallow	217	0.09
YOU Disallow	138	0.06
Youdao Bot Disallow	64	0.03
Google News Bot Disallow	47	0.02
Google API Bot Disallow	38	0.02
Google Video Bot Disallow	35	0.01
Google DuplexWeb Disallow	20	0.01
Google Favicon Disallow	19	0.01
Google FeedFetcher Disallow	2	0
Google Read Aloud Disallow	1	0
Google Web Light Disallow	1	0