Robots.txt Répartition des utilisations au Botswana

Distribution pour les sites Web utilisant Robots.txt technologies

Technologie	Sites Web	%
Robots Disallow	170	17,86
Google AdsBot Disallow	138	14,5
Googlebot Disallow	128	13,45
Google AdsBot Mobile Disallow	120	12,61
GPTBot Disallow	31	12,92
Ahrefs Bot Disallow	27	2,84
Common Crawl Bot Disallow	25	10,42
Anthropic Claude Bot Disallow	25	10,42
Google Bard Disallow	23	9,58
Facebook Bot Disallow	23	9,58
AppleBot Disallow	22	2,31
Cohere AI Disallow	22	9,17
Apple AI Bot Disallow	22	9,17
Perplexity	22	9,17
Yandex Bot Disallow	19	2
Google AdsBot Mobile Apps Disallow	16	1,68
Google Media Partners Disallow	13 ^*	1,37
Microsoft BingBot Disallow	13	1,37
Semrush Bot Disallow	13	1,37
Majestic Bot Disallow	12	1,26
ByteDance Bytespider Disallow	11	4,58
Yahoo Slurp Disallow	10	1,05
Soso Bot Disallow	9	0,95
Alexa Bot Disallow	8	3,33
Webz Omgili Disallow	7	2,92
Diffbot Disallow	6	2,5
YOU Disallow	6	2,5
Microsoft MSNBot Disallow	3	0,32
Google Image Bot Disallow	3	0,32
360 Bot Disallow	2	0,21
Exalead Bot Disallow	1	0,11
BuiltWithBot Disallow	1	0,11
DuckDuckGo Bot Disallow	1	0,11