Robots.txt Usage Distribution in Iran

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Majestic Bot Disallow	4,542	13.34
Ahrefs Bot Disallow	3,998	11.74
Semrush Bot Disallow	3,615	10.62
Robots Disallow	1,957	5.75
Microsoft BingBot Disallow	1,904	5.59
Common Crawl Bot Disallow	1,668	4.9
Microsoft MSNBot Disallow	1,626	4.78
Yandex Bot Disallow	1,611	4.73
Googlebot Disallow	1,575	4.63
Exalead Bot Disallow	1,555	4.57
Sogou Bot Disallow	1,337	3.93
Baidu Bot Disallow	1,312	3.85
ByteDance Bytespider Disallow	901	2.65
Some Robots Disallow	878	2.58
Yahoo Slurp Disallow	877	2.58
Soso Bot Disallow	698	2.05
GPTBot Disallow	572	1.68
Alexa Bot Disallow	474	1.39
360 Bot Disallow	403	1.18
BuiltWithBot Disallow	359	1.05
AppleBot Disallow	294	0.86
DuckDuckGo Bot Disallow	289	0.85
Google Media Partners Disallow	249 ^*	0.73
Google Image Bot Disallow	165	0.48
Google Bard Disallow	150	0.44
Anthropic Claude Bot Disallow	128	0.38
Google AdsBot Disallow	119	0.35
Google News Bot Disallow	95	0.28
Google Video Bot Disallow	94	0.28
Google AdsBot Mobile Disallow	84	0.25
Perplexity	64	0.19
Facebook Bot Disallow	62	0.18
Apple AI Bot Disallow	62	0.18
Cohere AI Disallow	58	0.17
Google AdsBot Mobile Apps Disallow	56	0.16
Webz Omgili Disallow	48	0.14
Google Favicon Disallow	33	0.1
Google API Bot Disallow	33	0.1
Google DuplexWeb Disallow	33	0.1
Diffbot Disallow	27	0.08
Pipl Disallow	19	0.06
YOU Disallow	15	0.04
Youdao Bot Disallow	5	0.01