Robots.txt Nutzungsverteilung In Finnland

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Google AdsBot Disallow	22.041	10,43
Google AdsBot Mobile Disallow	17.492	8,28
Googlebot Disallow	16.545	7,83
Robots Disallow	9.778	4,63
GPTBot Disallow	8.528	4,03
Common Crawl Bot Disallow	8.313	3,93
Anthropic Claude Bot Disallow	7.965	3,77
Ahrefs Bot Disallow	7.907	3,74
AppleBot Disallow	7.887	3,73
Google AdsBot Mobile Apps Disallow	7.867	3,72
Google Bard Disallow	7.809	3,69
Facebook Bot Disallow	7.710	3,65
Apple AI Bot Disallow	7.542	3,57
Perplexity	7.537	3,57
Cohere AI Disallow	7.506	3,55
Yandex Bot Disallow	6.811	3,22
Baidu Bot Disallow	6.773	3,2
Sogou Bot Disallow	4.492	2,13
Microsoft BingBot Disallow	4.460	2,11
Yahoo Slurp Disallow	4.345	2,06
Some Robots Disallow	4.224	2
Alexa Bot Disallow	3.933	1,86
Majestic Bot Disallow	3.131	1,48
Soso Bot Disallow	2.952	1,4
Semrush Bot Disallow	2.798	1,32
Google Media Partners Disallow	1.657 ^*	0,78
Google Image Bot Disallow	1.376	0,65
Microsoft MSNBot Disallow	1.254	0,59
DuckDuckGo Bot Disallow	1.108	0,52
Google Video Bot Disallow	1.074	0,51
Google News Bot Disallow	1.073	0,51
Google Favicon Disallow	1.065	0,5
Google API Bot Disallow	1.044	0,49
ByteDance Bytespider Disallow	1.043	0,49
Google DuplexWeb Disallow	1.041	0,49
Webz Omgili Disallow	914	0,43
Exalead Bot Disallow	762	0,36
Diffbot Disallow	702	0,33
YOU Disallow	631	0,3
360 Bot Disallow	229	0,11
BuiltWithBot Disallow	26	0,01
Pipl Disallow	11	0,01
Youdao Bot Disallow	9	0
Google FeedFetcher Disallow	1	0
Google Read Aloud Disallow	1	0