Robots.txt Nutzungsverteilung In El Salvador

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Google AdsBot Disallow	477	14,75
Googlebot Disallow	343	10,61
Google AdsBot Mobile Disallow	343	10,61
Robots Disallow	174	5,38
Ahrefs Bot Disallow	148	4,58
AppleBot Disallow	117	3,62
GPTBot Disallow	115	3,56
Common Crawl Bot Disallow	113	3,5
Anthropic Claude Bot Disallow	110	3,4
Google Bard Disallow	109	3,37
Facebook Bot Disallow	109	3,37
Apple AI Bot Disallow	109	3,37
Perplexity	109	3,37
Cohere AI Disallow	107	3,31
Google AdsBot Mobile Apps Disallow	107	3,31
Google Media Partners Disallow	89 ^*	2,75
Baidu Bot Disallow	85	2,63
Some Robots Disallow	64	1,98
Yandex Bot Disallow	61	1,89
Microsoft BingBot Disallow	54	1,67
Sogou Bot Disallow	51	1,58
Alexa Bot Disallow	51	1,58
Yahoo Slurp Disallow	49	1,52
Soso Bot Disallow	46	1,42
Semrush Bot Disallow	27	0,84
Majestic Bot Disallow	26	0,8
Microsoft MSNBot Disallow	15	0,46
Exalead Bot Disallow	6	0,19
ByteDance Bytespider Disallow	4	0,12
Google Image Bot Disallow	4	0,12
Webz Omgili Disallow	3	0,09
360 Bot Disallow	3	0,09
BuiltWithBot Disallow	2	0,06
Diffbot Disallow	1	0,03
Youdao Bot Disallow	1	0,03
Pipl Disallow	1	0,03