Robots.txt Nutzungsverteilung In Ungarn

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Majestic Bot Disallow	12.175	8,45
Robots Disallow	10.015	6,95
Ahrefs Bot Disallow	9.807	6,8
Googlebot Disallow	9.631	6,68
Google AdsBot Disallow	9.437	6,55
Google AdsBot Mobile Disallow	6.702	4,65
Semrush Bot Disallow	6.203	4,3
GPTBot Disallow	5.433	3,77
Common Crawl Bot Disallow	5.229	3,63
Anthropic Claude Bot Disallow	5.106	3,54
Yandex Bot Disallow	4.779	3,32
AppleBot Disallow	4.723	3,28
Baidu Bot Disallow	4.252	2,95
Google Bard Disallow	4.057	2,82
Facebook Bot Disallow	3.980	2,76
ByteDance Bytespider Disallow	3.890	2,7
Apple AI Bot Disallow	3.879	2,69
Cohere AI Disallow	3.810	2,64
Perplexity	3.804	2,64
Webz Omgili Disallow	3.103	2,15
Alexa Bot Disallow	3.087	2,14
Diffbot Disallow	2.884	2
YOU Disallow	2.826	1,96
Microsoft BingBot Disallow	2.532	1,76
Some Robots Disallow	2.023	1,4
Sogou Bot Disallow	1.657	1,15
Exalead Bot Disallow	1.609	1,12
Yahoo Slurp Disallow	1.379	0,96
Soso Bot Disallow	1.305	0,91
Google Media Partners Disallow	1.082 ^*	0,75
Google AdsBot Mobile Apps Disallow	1.023	0,71
Google Image Bot Disallow	665	0,46
Microsoft MSNBot Disallow	527	0,37
360 Bot Disallow	302	0,21
Google News Bot Disallow	243	0,17
BuiltWithBot Disallow	237	0,16
Google Video Bot Disallow	216	0,15
DuckDuckGo Bot Disallow	145	0,1
Google Favicon Disallow	110	0,08
Google API Bot Disallow	105	0,07
Google DuplexWeb Disallow	104	0,07
Google FeedFetcher Disallow	19	0,01
Pipl Disallow	17	0,01
Youdao Bot Disallow	4	0