Robots.txt Répartition des utilisations aux Pays-Bas

Distribution pour les sites Web utilisant Robots.txt technologies

Technologie	Sites Web	%
Googlebot Disallow	113 368	11,79
Google AdsBot Disallow	99 197	10,32
Baidu Bot Disallow	70 516	7,34
Google AdsBot Mobile Disallow	61 277	6,37
Robots Disallow	57 414	5,97
Ahrefs Bot Disallow	54 167	5,63
Sogou Bot Disallow	46 746	4,86
Some Robots Disallow	39 214	4,08
Yandex Bot Disallow	37 184	3,87
Majestic Bot Disallow	36 166	3,76
Microsoft BingBot Disallow	25 450	2,65
Alexa Bot Disallow	25 197	2,62
Yahoo Slurp Disallow	24 494	2,55
Semrush Bot Disallow	23 872	2,48
Soso Bot Disallow	21 563	2,24
GPTBot Disallow	20 764	2,16
AppleBot Disallow	20 215	2,1
Common Crawl Bot Disallow	19 573	2,04
Anthropic Claude Bot Disallow	18 730	1,95
Google Bard Disallow	17 407	1,81
Facebook Bot Disallow	16 640	1,73
Apple AI Bot Disallow	15 634	1,63
Perplexity	15 354	1,6
Cohere AI Disallow	15 234	1,58
Google AdsBot Mobile Apps Disallow	13 575	1,41
ByteDance Bytespider Disallow	8 627	0,9
Exalead Bot Disallow	8 099	0,84
Microsoft MSNBot Disallow	5 641	0,59
Google Image Bot Disallow	5 443	0,57
Google Media Partners Disallow	4 947 ^*	0,51
Webz Omgili Disallow	3 972	0,41
Diffbot Disallow	2 713	0,28
360 Bot Disallow	2 075	0,22
YOU Disallow	2 034	0,21
DuckDuckGo Bot Disallow	1 803	0,19
Google News Bot Disallow	1 238	0,13
BuiltWithBot Disallow	1 148	0,12
Google Video Bot Disallow	1 142	0,12
Google Favicon Disallow	1 080	0,11
Google API Bot Disallow	986	0,1
Google DuplexWeb Disallow	979	0,1
Pipl Disallow	324	0,03
Youdao Bot Disallow	57	0,01
Google FeedFetcher Disallow	20	0
Google Read Aloud Disallow	7	0
Google Web Light Disallow	4	0
LLMS Text	1	0