Robots.txt Répartition de l'utilisation dans Afrique du Sud

Distribution pour les sites Web utilisant Robots.txt les technologies

Technologie	Sites Internet	%
Googlebot Disallow	39 188	14,4
Google AdsBot Disallow	35 553	13,07
Google AdsBot Mobile Disallow	22 276	8,19
Baidu Bot Disallow	20 836	7,66
Ahrefs Bot Disallow	14 318	5,26
Yandex Bot Disallow	11 712	4,3
Sogou Bot Disallow	11 446	4,21
Microsoft BingBot Disallow	11 257	4,14
Some Robots Disallow	11 053	4,06
Yahoo Slurp Disallow	10 604	3,9
Alexa Bot Disallow	10 499	3,86
Soso Bot Disallow	10 112	3,72
Robots Disallow	8 086	2,97
GPTBot Disallow	4 777	1,76
Anthropic Claude Bot Disallow	4 756	1,75
Common Crawl Bot Disallow	4 401	1,62
AppleBot Disallow	4 293	1,58
Google Bard Disallow	4 235	1,56
Facebook Bot Disallow	4 162	1,53
Apple AI Bot Disallow	4 066	1,49
Perplexity	4 044	1,49
Cohere AI Disallow	4 014	1,48
Google AdsBot Mobile Apps Disallow	3 828	1,41
Majestic Bot Disallow	2 969	1,09
Semrush Bot Disallow	2 626	0,97
ByteDance Bytespider Disallow	1 985	0,73
Google Media Partners Disallow	1 161 ^*	0,43
Microsoft MSNBot Disallow	734	0,27
Google Image Bot Disallow	465	0,17
Webz Omgili Disallow	440	0,16
360 Bot Disallow	381	0,14
Diffbot Disallow	360	0,13
DuckDuckGo Bot Disallow	299	0,11
Exalead Bot Disallow	273	0,1
YOU Disallow	262	0,1
Google News Bot Disallow	116	0,04
Google Video Bot Disallow	109	0,04
Google Favicon Disallow	93	0,03
Google API Bot Disallow	75	0,03
Google DuplexWeb Disallow	74	0,03
BuiltWithBot Disallow	64	0,02
Pipl Disallow	43	0,02
Youdao Bot Disallow	12	0
Google FeedFetcher Disallow	1	0