Robots.txt Répartition de l'utilisation dans Oman

Distribution pour les sites Web utilisant Robots.txt les technologies

Technologie	Sites Internet	%
Google AdsBot Disallow	453	15,77
Googlebot Disallow	416	14,48
Google AdsBot Mobile Disallow	350	12,19
Robots Disallow	190	6,62
Ahrefs Bot Disallow	127	4,42
Baidu Bot Disallow	97	3,38
Common Crawl Bot Disallow	74	2,58
Yandex Bot Disallow	70	2,44
GPTBot Disallow	70	2,44
Google Bard Disallow	65	2,26
Sogou Bot Disallow	64	2,23
AppleBot Disallow	64	2,23
Anthropic Claude Bot Disallow	61	2,12
Perplexity	61	2,12
Facebook Bot Disallow	60	2,09
Apple AI Bot Disallow	60	2,09
Yahoo Slurp Disallow	59	2,05
Soso Bot Disallow	59	2,05
Cohere AI Disallow	59	2,05
Google AdsBot Mobile Apps Disallow	59	2,05
Some Robots Disallow	58	2,02
Alexa Bot Disallow	57	1,98
Microsoft BingBot Disallow	53	1,85
Google Media Partners Disallow	45 ^*	1,57
Majestic Bot Disallow	43	1,5
Semrush Bot Disallow	30	1,04
Exalead Bot Disallow	13	0,45
360 Bot Disallow	12	0,42
Microsoft MSNBot Disallow	11	0,38
BuiltWithBot Disallow	10	0,35
ByteDance Bytespider Disallow	5	0,17
Google Image Bot Disallow	5	0,17
DuckDuckGo Bot Disallow	3	0,1
Webz Omgili Disallow	2	0,07
Google News Bot Disallow	2	0,07
Youdao Bot Disallow	2	0,07
Google Video Bot Disallow	1	0,03
Google Favicon Disallow	1	0,03
YOU Disallow	1	0,03