Robots.txt Usage Distribution in Kazakhstan

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Yandex Bot Disallow	3,051	24.31
Robots Disallow	2,726	21.72
Googlebot Disallow	1,642	13.08
Semrush Bot Disallow	740	5.9
Majestic Bot Disallow	686	5.47
Ahrefs Bot Disallow	629	5.01
Microsoft MSNBot Disallow	343	2.73
Microsoft BingBot Disallow	318	2.53
GPTBot Disallow	272	20.96
Some Robots Disallow	262	2.09
Google AdsBot Disallow	256	2.04
Google AdsBot Mobile Disallow	202	1.61
Alexa Bot Disallow	202	15.56
Common Crawl Bot Disallow	178	13.71
Google Media Partners Disallow	133 ^*	1.06
Baidu Bot Disallow	117	0.93
Anthropic Claude Bot Disallow	113	8.71
Google Bard Disallow	103	7.94
Yahoo Slurp Disallow	96	0.76
Sogou Bot Disallow	91	0.73
Webz Omgili Disallow	84	6.47
Google Image Bot Disallow	52	0.41
Soso Bot Disallow	51	0.41
Exalead Bot Disallow	33	0.26
AppleBot Disallow	29	0.23
ByteDance Bytespider Disallow	24	1.85
Facebook Bot Disallow	17	1.31
Apple AI Bot Disallow	13	1
Perplexity	13	1
360 Bot Disallow	12	0.1
Cohere AI Disallow	11	0.85
Google AdsBot Mobile Apps Disallow	10	0.08
Google News Bot Disallow	8	0.06
Google Video Bot Disallow	5	0.04
DuckDuckGo Bot Disallow	5	0.04
BuiltWithBot Disallow	4	0.03
Google Favicon Disallow	4	0.03
Google API Bot Disallow	4	0.03
Google DuplexWeb Disallow	4	0.03
Diffbot Disallow	3	0.23
YOU Disallow	1	0.08
Youdao Bot Disallow	1	0.01
Pipl Disallow	1	0.01
Google FeedFetcher Disallow	1	0.01
Google Read Aloud Disallow	1	0.01