Robots.txt Répartition des utilisations à l'île Norfolk

Distribution pour les sites Web utilisant Robots.txt technologies

Technologie	Sites Web	%
Google Media Partners Disallow	13 ^*	12,75
Robots Disallow	9	8,82
Googlebot Disallow	8	7,84
Baidu Bot Disallow	7	6,86
Google AdsBot Disallow	4	3,92
Yandex Bot Disallow	4	3,92
Microsoft BingBot Disallow	4	3,92
Yahoo Slurp Disallow	4	3,92
Google AdsBot Mobile Disallow	3	2,94
Alexa Bot Disallow	3	2,94
Soso Bot Disallow	3	2,94
Ahrefs Bot Disallow	3	2,94
Common Crawl Bot Disallow	3	2,94
AppleBot Disallow	3	2,94
Anthropic Claude Bot Disallow	3	2,94
Google Bard Disallow	3	2,94
Facebook Bot Disallow	3	2,94
Apple AI Bot Disallow	3	2,94
Google AdsBot Mobile Apps Disallow	3	2,94
GPTBot Disallow	2	1,96
Cohere AI Disallow	2	1,96
Perplexity	2	1,96
Semrush Bot Disallow	1	0,98
Microsoft MSNBot Disallow	1	0,98
Webz Omgili Disallow	1	0,98
Google Image Bot Disallow	1	0,98
Google News Bot Disallow	1	0,98
Google Video Bot Disallow	1	0,98
DuckDuckGo Bot Disallow	1	0,98
Google Favicon Disallow	1	0,98
Google API Bot Disallow	1	0,98
Google DuplexWeb Disallow	1	0,98