Robots.txt Nutzungsverteilung in Libyen

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Google Media Partners Disallow	80 ^*	37,38
Robots Disallow	38	17,76
Ahrefs Bot Disallow	8	3,74
GPTBot Disallow	7	11,86
Common Crawl Bot Disallow	7	11,86
Anthropic Claude Bot Disallow	7	11,86
Google Bard Disallow	7	11,86
Facebook Bot Disallow	7	11,86
Perplexity	7	11,86
AppleBot Disallow	6	2,8
Cohere AI Disallow	6	10,17
Apple AI Bot Disallow	6	10,17
Google AdsBot Mobile Apps Disallow	6	2,8
Yandex Bot Disallow	5	2,34
Microsoft BingBot Disallow	3	1,4
Alexa Bot Disallow	3	5,08
Yahoo Slurp Disallow	3	1,4
Soso Bot Disallow	3	1,4
Semrush Bot Disallow	2	0,93
Majestic Bot Disallow	1	0,47
ByteDance Bytespider Disallow	1	1,69
Google Image Bot Disallow	1	0,47