Robots.txt Usage Distribution in Azerbaijan

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	689	21.48
Robots Disallow	601	18.73
Google AdsBot Disallow	393	12.25
Google AdsBot Mobile Disallow	353	11
Yandex Bot Disallow	235	7.33
Some Robots Disallow	104	3.24
Ahrefs Bot Disallow	98	3.05
Baidu Bot Disallow	80	2.49
Google Media Partners Disallow	73 ^*	2.28
Sogou Bot Disallow	55	1.71
Semrush Bot Disallow	53	1.65
Majestic Bot Disallow	52	1.62
GPTBot Disallow	43	1.34
Microsoft BingBot Disallow	34	1.06
Common Crawl Bot Disallow	34	1.06
Alexa Bot Disallow	33	1.03
Yahoo Slurp Disallow	32	1
Microsoft MSNBot Disallow	30	0.94
Soso Bot Disallow	26	0.81
Anthropic Claude Bot Disallow	20	0.62
Google Bard Disallow	20	0.62
AppleBot Disallow	17	0.53
Facebook Bot Disallow	14	0.44
Exalead Bot Disallow	12	0.37
Cohere AI Disallow	11	0.34
Apple AI Bot Disallow	11	0.34
ByteDance Bytespider Disallow	11	0.34
360 Bot Disallow	11	0.34
Perplexity	10	0.31
Google AdsBot Mobile Apps Disallow	10	0.31
Google Image Bot Disallow	9	0.28
Webz Omgili Disallow	8	0.25
BuiltWithBot Disallow	6	0.19
Google News Bot Disallow	3	0.09
Google Video Bot Disallow	3	0.09
DuckDuckGo Bot Disallow	3	0.09
Google Favicon Disallow	3	0.09
Google API Bot Disallow	3	0.09
Google DuplexWeb Disallow	3	0.09
Diffbot Disallow	1	0.03
LLMS Text	1	0.03