Robots.txt Répartition de l'utilisation dans L'Iran

Distribution pour les sites Web utilisant Robots.txt les technologies

Technologie	Sites Internet	%
Majestic Bot Disallow	5 130	13,81
Ahrefs Bot Disallow	4 492	12,09
Semrush Bot Disallow	4 064	10,94
Common Crawl Bot Disallow	2 497	6,72
Exalead Bot Disallow	2 425	6,53
Microsoft BingBot Disallow	2 327	6,26
Microsoft MSNBot Disallow	2 090	5,63
Robots Disallow	1 468	3,95
Googlebot Disallow	1 460	3,93
Baidu Bot Disallow	1 283	3,45
Sogou Bot Disallow	1 231	3,31
Yandex Bot Disallow	1 181	3,18
Soso Bot Disallow	1 027	2,76
Yahoo Slurp Disallow	902	2,43
360 Bot Disallow	764	2,06
BuiltWithBot Disallow	731	1,97
GPTBot Disallow	619	1,67
Some Robots Disallow	585	1,57
Alexa Bot Disallow	489	1,32
AppleBot Disallow	372	1
DuckDuckGo Bot Disallow	295	0,79
Google Image Bot Disallow	189	0,51
Google Media Partners Disallow	179 ^*	0,48
Google Bard Disallow	160	0,43
Google AdsBot Disallow	133	0,36
Anthropic Claude Bot Disallow	128	0,34
Google News Bot Disallow	125	0,34
Google Video Bot Disallow	123	0,33
Google AdsBot Mobile Disallow	97	0,26
ByteDance Bytespider Disallow	85	0,23
Perplexity	69	0,19
Cohere AI Disallow	63	0,17
Apple AI Bot Disallow	62	0,17
Facebook Bot Disallow	61	0,16
Google AdsBot Mobile Apps Disallow	55	0,15
Webz Omgili Disallow	49	0,13
Google Favicon Disallow	27	0,07
Google API Bot Disallow	27	0,07
Google DuplexWeb Disallow	27	0,07
Diffbot Disallow	23	0,06
Pipl Disallow	21	0,06
YOU Disallow	15	0,04
Youdao Bot Disallow	2	0,01