Robots.txt Usage Distribution in Iran

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Majestic Bot Disallow	3,520	12.91
Ahrefs Bot Disallow	3,111	11.41
Semrush Bot Disallow	2,828	10.37
Robots Disallow	1,653	6.06
Microsoft BingBot Disallow	1,575	5.78
Yandex Bot Disallow	1,485	5.45
Microsoft MSNBot Disallow	1,338	4.91
Common Crawl Bot Disallow	1,280	34.35
Googlebot Disallow	1,235	4.53
Exalead Bot Disallow	1,175	4.31
Some Robots Disallow	1,054	3.87
Baidu Bot Disallow	1,002	3.68
ByteDance Bytespider Disallow	927	24.88
Yahoo Slurp Disallow	773	2.84
Sogou Bot Disallow	767	2.81
Soso Bot Disallow	494	1.81
Alexa Bot Disallow	430	11.54
GPTBot Disallow	381	10.23
Google Media Partners Disallow	292 ^*	1.07
DuckDuckGo Bot Disallow	248	0.91
AppleBot Disallow	213	0.78
360 Bot Disallow	209	0.77
BuiltWithBot Disallow	177	0.65
Google Bard Disallow	127	3.41
Google Image Bot Disallow	118	0.43
Google AdsBot Disallow	113	0.41
Anthropic Claude Bot Disallow	101	2.71
Google AdsBot Mobile Disallow	80	0.29
Google AdsBot Mobile Apps Disallow	57	0.21
Google News Bot Disallow	54	0.2
Google Video Bot Disallow	54	0.2
Facebook Bot Disallow	51	1.37
Perplexity	51	1.37
Cohere AI Disallow	46	1.23
Apple AI Bot Disallow	45	1.21
Google Favicon Disallow	37	0.14
Google API Bot Disallow	37	0.14
Google DuplexWeb Disallow	37	0.14
Webz Omgili Disallow	32	0.86
Diffbot Disallow	22	0.59
Pipl Disallow	17	0.06
YOU Disallow	12	0.32
Youdao Bot Disallow	3	0.01
LLMS Text	1	0