Robots.txt Nutzungsverteilung in Namibia

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Google AdsBot Disallow	360	18,06
Googlebot Disallow	356	17,86
Google AdsBot Mobile Disallow	327	16,41
Robots Disallow	136	6,82
Common Crawl Bot Disallow	47	11,99
Yandex Bot Disallow	45	2,26
Ahrefs Bot Disallow	45	2,26
GPTBot Disallow	45	11,48
Some Robots Disallow	44	2,21
AppleBot Disallow	42	2,11
Google Bard Disallow	42	10,71
Microsoft BingBot Disallow	40	2,01
Alexa Bot Disallow	40	10,2
Anthropic Claude Bot Disallow	40	10,2
Facebook Bot Disallow	40	10,2
Apple AI Bot Disallow	40	10,2
Google Media Partners Disallow	40 ^*	2,01
Cohere AI Disallow	39	9,95
Perplexity	38	9,69
Yahoo Slurp Disallow	36	1,81
Google AdsBot Mobile Apps Disallow	36	1,81
Soso Bot Disallow	31	1,56
Majestic Bot Disallow	19	0,95
Semrush Bot Disallow	18	0,9
Microsoft MSNBot Disallow	9	0,45
Google Image Bot Disallow	8	0,4
Exalead Bot Disallow	7	0,35
ByteDance Bytespider Disallow	6	1,53
Webz Omgili Disallow	4	1,02
Diffbot Disallow	3	0,77
360 Bot Disallow	2	0,1
BuiltWithBot Disallow	2	0,1
YOU Disallow	2	0,51
Google News Bot Disallow	1	0,05
Google Video Bot Disallow	1	0,05
DuckDuckGo Bot Disallow	1	0,05
Google Favicon Disallow	1	0,05