Robots.txt Nutzungsverteilung In Portugal

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Google AdsBot Disallow	21.694	14,68
Googlebot Disallow	19.555	13,23
Google AdsBot Mobile Disallow	16.446	11,13
Ahrefs Bot Disallow	7.782	5,27
Robots Disallow	5.931	4,01
Yandex Bot Disallow	5.373	3,64
GPTBot Disallow	4.318	2,92
Baidu Bot Disallow	4.237	2,87
Common Crawl Bot Disallow	4.052	2,74
Anthropic Claude Bot Disallow	3.950	2,67
Google Bard Disallow	3.904	2,64
Facebook Bot Disallow	3.823	2,59
AppleBot Disallow	3.800	2,57
Apple AI Bot Disallow	3.625	2,45
Google AdsBot Mobile Apps Disallow	3.577	2,42
Perplexity	3.551	2,4
Cohere AI Disallow	3.542	2,4
Yahoo Slurp Disallow	3.304	2,24
Microsoft BingBot Disallow	3.155	2,13
Alexa Bot Disallow	3.069	2,08
Sogou Bot Disallow	2.995	2,03
Majestic Bot Disallow	2.944	1,99
Some Robots Disallow	2.814	1,9
Soso Bot Disallow	2.526	1,71
Semrush Bot Disallow	2.458	1,66
Google Media Partners Disallow	1.265 ^*	0,86
Microsoft MSNBot Disallow	1.140	0,77
Exalead Bot Disallow	525	0,36
ByteDance Bytespider Disallow	507	0,34
Google Image Bot Disallow	494	0,33
Webz Omgili Disallow	303	0,21
360 Bot Disallow	205	0,14
Google News Bot Disallow	117	0,08
DuckDuckGo Bot Disallow	116	0,08
Google Video Bot Disallow	110	0,07
Google Favicon Disallow	99	0,07
BuiltWithBot Disallow	98	0,07
Google API Bot Disallow	89	0,06
Google DuplexWeb Disallow	89	0,06
Diffbot Disallow	84	0,06
YOU Disallow	58	0,04
Pipl Disallow	36	0,02
Youdao Bot Disallow	30	0,02
Google Web Light Disallow	2	0
Google Read Aloud Disallow	1	0