Robots.txt Répartition des utilisations en Australie

Distribution pour les sites Web utilisant Robots.txt technologies

Technologie	Sites Web	%
Google AdsBot Disallow	351 068	12,95
Google AdsBot Mobile Disallow	251 411	9,27
Googlebot Disallow	246 323	9,09
Baidu Bot Disallow	155 939	5,75
Microsoft BingBot Disallow	114 117	4,21
Sogou Bot Disallow	102 395	3,78
Common Crawl Bot Disallow	99 569	3,67
GPTBot Disallow	99 489	3,67
AppleBot Disallow	97 955	3,61
Anthropic Claude Bot Disallow	97 762	3,61
Google Bard Disallow	97 504	3,6
Facebook Bot Disallow	97 092	3,58
Apple AI Bot Disallow	96 391	3,56
Cohere AI Disallow	95 950	3,54
Perplexity	95 856	3,54
Google AdsBot Mobile Apps Disallow	95 126	3,51
Ahrefs Bot Disallow	84 040	3,1
Yandex Bot Disallow	76 581	2,82
Alexa Bot Disallow	74 976	2,77
Yahoo Slurp Disallow	73 829	2,72
Soso Bot Disallow	71 600	2,64
Some Robots Disallow	59 089	2,18
Robots Disallow	33 722	1,24
Majestic Bot Disallow	10 966	0,4
Semrush Bot Disallow	9 026	0,33
Google Media Partners Disallow	4 069 ^*	0,15
Microsoft MSNBot Disallow	3 419	0,13
ByteDance Bytespider Disallow	3 047	0,11
Exalead Bot Disallow	2 783	0,1
Webz Omgili Disallow	2 291	0,08
Google Image Bot Disallow	1 965	0,07
Diffbot Disallow	1 097	0,04
360 Bot Disallow	1 047	0,04
Google News Bot Disallow	521	0,02
Pipl Disallow	512	0,02
Google Video Bot Disallow	455	0,02
DuckDuckGo Bot Disallow	451	0,02
BuiltWithBot Disallow	382	0,01
Google Favicon Disallow	377	0,01
YOU Disallow	304	0,01
Google API Bot Disallow	238	0,01
Google DuplexWeb Disallow	226	0,01
Youdao Bot Disallow	123	0
Google FeedFetcher Disallow	12	0
Google Web Light Disallow	3	0
Google Read Aloud Disallow	2	0
LLMS Text	2	0