Robots.txt Nutzungsverteilung In Guernsey

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Googlebot Disallow	1.911	14
Baidu Bot Disallow	1.564	11,46
Some Robots Disallow	1.402	10,27
Sogou Bot Disallow	1.074	7,87
Google AdsBot Disallow	798	5,85
Robots Disallow	767	5,62
Google AdsBot Mobile Disallow	515	3,77
Yandex Bot Disallow	410	3
Google Media Partners Disallow	406 ^*	2,97
Microsoft BingBot Disallow	389	2,85
Alexa Bot Disallow	387	2,84
Yahoo Slurp Disallow	385	2,82
Ahrefs Bot Disallow	377	2,76
Soso Bot Disallow	373	2,73
GPTBot Disallow	297	2,18
Common Crawl Bot Disallow	257	1,88
Anthropic Claude Bot Disallow	253	1,85
Google Bard Disallow	253	1,85
AppleBot Disallow	248	1,82
Facebook Bot Disallow	244	1,79
Apple AI Bot Disallow	235	1,72
Perplexity	234	1,71
Google AdsBot Mobile Apps Disallow	228	1,67
Cohere AI Disallow	226	1,66
Majestic Bot Disallow	124	0,91
Semrush Bot Disallow	77	0,56
ByteDance Bytespider Disallow	26	0,19
Google Image Bot Disallow	25	0,18
Webz Omgili Disallow	22	0,16
Microsoft MSNBot Disallow	18	0,13
Exalead Bot Disallow	17	0,12
360 Bot Disallow	13	0,1
Google News Bot Disallow	12	0,09
DuckDuckGo Bot Disallow	12	0,09
Google Video Bot Disallow	11	0,08
Google Favicon Disallow	11	0,08
Diffbot Disallow	10	0,07
Google API Bot Disallow	10	0,07
Google DuplexWeb Disallow	10	0,07
Pipl Disallow	7	0,05
YOU Disallow	6	0,04
BuiltWithBot Disallow	5	0,04