Robots.txt Répartition des utilisations en Tunisie

Distribution pour les sites Web utilisant Robots.txt technologies

Technologie	Sites Web	%
Robots Disallow	474	13,81
Googlebot Disallow	351	10,23
Google AdsBot Disallow	286	8,33
Ahrefs Bot Disallow	210	6,12
Baidu Bot Disallow	186	5,42
Sogou Bot Disallow	184	5,36
Yandex Bot Disallow	179	5,22
Microsoft BingBot Disallow	154	4,49
Soso Bot Disallow	145	4,22
Google AdsBot Mobile Disallow	134	3,9
Alexa Bot Disallow	132	3,85
Yahoo Slurp Disallow	127	3,7
Some Robots Disallow	93	2,71
Majestic Bot Disallow	81	2,36
Semrush Bot Disallow	67	1,95
GPTBot Disallow	59	1,72
Common Crawl Bot Disallow	58	1,69
Anthropic Claude Bot Disallow	49	1,43
Facebook Bot Disallow	46	1,34
Cohere AI Disallow	46	1,34
Microsoft MSNBot Disallow	42	1,22
Exalead Bot Disallow	40	1,17
ByteDance Bytespider Disallow	30	0,87
360 Bot Disallow	30	0,87
Webz Omgili Disallow	29	0,84
BuiltWithBot Disallow	27	0,79
Google AdsBot Mobile Apps Disallow	25	0,73
Google Bard Disallow	21	0,61
AppleBot Disallow	20	0,58
Perplexity	19	0,55
Apple AI Bot Disallow	18	0,52
Google Image Bot Disallow	15	0,44
Google Media Partners Disallow	15 ^*	0,44
Google News Bot Disallow	7	0,2
Google Video Bot Disallow	6	0,17
DuckDuckGo Bot Disallow	6	0,17
Google Favicon Disallow	6	0,17
Google API Bot Disallow	6	0,17
Google DuplexWeb Disallow	6	0,17
Diffbot Disallow	1	0,03
YOU Disallow	1	0,03
Youdao Bot Disallow	1	0,03