Robots.txt Nutzungsverteilung In Neu-Kaledonien

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Googlebot Disallow	320	19,83
Google AdsBot Disallow	286	17,72
Google AdsBot Mobile Disallow	250	15,49
Robots Disallow	128	7,93
Ahrefs Bot Disallow	70	4,34
Google Image Bot Disallow	68	4,21
Some Robots Disallow	65	4,03
Semrush Bot Disallow	48	2,97
Google Media Partners Disallow	45 ^*	2,79
Majestic Bot Disallow	29	1,8
Webz Omgili Disallow	26	1,61
Baidu Bot Disallow	19	1,18
Microsoft BingBot Disallow	18	1,12
Yahoo Slurp Disallow	18	1,12
DuckDuckGo Bot Disallow	18	1,12
Alexa Bot Disallow	16	0,99
Google AdsBot Mobile Apps Disallow	16	0,99
Yandex Bot Disallow	14	0,87
Google News Bot Disallow	12	0,74
Common Crawl Bot Disallow	11	0,68
Microsoft MSNBot Disallow	11	0,68
Google Video Bot Disallow	11	0,68
Google Favicon Disallow	11	0,68
Google API Bot Disallow	11	0,68
Google DuplexWeb Disallow	11	0,68
GPTBot Disallow	10	0,62
Google Bard Disallow	10	0,62
AppleBot Disallow	9	0,56
Anthropic Claude Bot Disallow	9	0,56
Facebook Bot Disallow	9	0,56
Sogou Bot Disallow	8	0,5
Apple AI Bot Disallow	8	0,5
Perplexity	7	0,43
Cohere AI Disallow	5	0,31
ByteDance Bytespider Disallow	4	0,25
Soso Bot Disallow	2	0,12
Exalead Bot Disallow	1	0,06