Robots.txt Usage Distribution in Kazakhstan

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Yandex Bot Disallow	3,178	24.05
Robots Disallow	2,611	19.76
Googlebot Disallow	2,128	16.11
Semrush Bot Disallow	745	5.64
Majestic Bot Disallow	694	5.25
Ahrefs Bot Disallow	639	4.84
Microsoft MSNBot Disallow	345	2.61
Google AdsBot Disallow	313	2.37
Microsoft BingBot Disallow	313	2.37
Some Robots Disallow	292	2.21
GPTBot Disallow	251	1.9
Google AdsBot Mobile Disallow	245	1.85
Alexa Bot Disallow	201	1.52
Common Crawl Bot Disallow	168	1.27
Google Media Partners Disallow	161 ^*	1.22
Baidu Bot Disallow	133	1.01
Anthropic Claude Bot Disallow	114	0.86
Yahoo Slurp Disallow	103	0.78
Google Bard Disallow	103	0.78
Sogou Bot Disallow	86	0.65
Webz Omgili Disallow	85	0.64
Soso Bot Disallow	52	0.39
Google Image Bot Disallow	51	0.39
Exalead Bot Disallow	32	0.24
AppleBot Disallow	29	0.22
ByteDance Bytespider Disallow	24	0.18
Facebook Bot Disallow	18	0.14
Apple AI Bot Disallow	13	0.1
Cohere AI Disallow	12	0.09
Google AdsBot Mobile Apps Disallow	11	0.08
360 Bot Disallow	11	0.08
Perplexity	10	0.08
Google News Bot Disallow	9	0.07
Google Video Bot Disallow	5	0.04
DuckDuckGo Bot Disallow	5	0.04
Google Favicon Disallow	4	0.03
Google API Bot Disallow	4	0.03
Google DuplexWeb Disallow	4	0.03
Diffbot Disallow	3	0.02
BuiltWithBot Disallow	2	0.02
YOU Disallow	1	0.01
Youdao Bot Disallow	1	0.01
Pipl Disallow	1	0.01
Google FeedFetcher Disallow	1	0.01
Google Read Aloud Disallow	1	0.01