Robots.txt Nutzungsverteilung In Luxemburg

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Googlebot Disallow	8.323	14,01
Google AdsBot Disallow	4.514	7,6
Baidu Bot Disallow	4.394	7,4
Yandex Bot Disallow	4.143	6,98
Microsoft BingBot Disallow	3.989	6,72
Google AdsBot Mobile Disallow	3.985	6,71
Yahoo Slurp Disallow	3.971	6,69
Alexa Bot Disallow	3.970	6,68
Sogou Bot Disallow	3.957	6,66
Soso Bot Disallow	3.883	6,54
Robots Disallow	2.003	3,37
Majestic Bot Disallow	1.482	2,5
Ahrefs Bot Disallow	1.173	1,97
GPTBot Disallow	864	1,45
Semrush Bot Disallow	751	1,26
Common Crawl Bot Disallow	700	1,18
Anthropic Claude Bot Disallow	679	1,14
Google Bard Disallow	665	1,12
Google AdsBot Mobile Apps Disallow	659	1,11
Facebook Bot Disallow	629	1,06
AppleBot Disallow	626	1,05
Cohere AI Disallow	603	1,02
Perplexity	602	1,01
Apple AI Bot Disallow	601	1,01
Some Robots Disallow	558	0,94
Google Media Partners Disallow	372 ^*	0,63
Google Image Bot Disallow	177	0,3
Microsoft MSNBot Disallow	173	0,29
Google News Bot Disallow	101	0,17
Google Video Bot Disallow	98	0,16
DuckDuckGo Bot Disallow	98	0,16
Google Favicon Disallow	93	0,16
Google API Bot Disallow	88	0,15
Google DuplexWeb Disallow	88	0,15
ByteDance Bytespider Disallow	87	0,15
Exalead Bot Disallow	86	0,14
Webz Omgili Disallow	83	0,14
360 Bot Disallow	45	0,08
Diffbot Disallow	30	0,05
BuiltWithBot Disallow	22	0,04
YOU Disallow	10	0,02
Youdao Bot Disallow	9	0,02
Pipl Disallow	9	0,02
Google Read Aloud Disallow	2	0
Google Web Light Disallow	1	0