Robots.txt Nutzungsverteilung In Georgia

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Googlebot Disallow	1.376	16,95
Google AdsBot Disallow	1.234	15,2
Google AdsBot Mobile Disallow	984	12,12
Robots Disallow	612	7,54
Yandex Bot Disallow	381	4,69
Some Robots Disallow	355	4,37
Baidu Bot Disallow	323	3,98
Ahrefs Bot Disallow	290	3,57
Sogou Bot Disallow	217	2,67
Alexa Bot Disallow	196	2,41
Yahoo Slurp Disallow	196	2,41
Soso Bot Disallow	195	2,4
Microsoft BingBot Disallow	190	2,34
Common Crawl Bot Disallow	136	1,68
GPTBot Disallow	119	1,47
Anthropic Claude Bot Disallow	111	1,37
AppleBot Disallow	107	1,32
Google Bard Disallow	107	1,32
Facebook Bot Disallow	102	1,26
Apple AI Bot Disallow	100	1,23
Semrush Bot Disallow	98	1,21
Google AdsBot Mobile Apps Disallow	95	1,17
Perplexity	93	1,15
Majestic Bot Disallow	92	1,13
Cohere AI Disallow	91	1,12
Google Media Partners Disallow	89 ^*	1,1
Microsoft MSNBot Disallow	37	0,46
Exalead Bot Disallow	36	0,44
360 Bot Disallow	32	0,39
BuiltWithBot Disallow	30	0,37
Google Image Bot Disallow	26	0,32
Webz Omgili Disallow	12	0,15
ByteDance Bytespider Disallow	11	0,14
Google News Bot Disallow	8	0,1
DuckDuckGo Bot Disallow	8	0,1
Google Video Bot Disallow	7	0,09
Google Favicon Disallow	6	0,07
Google API Bot Disallow	6	0,07
Google DuplexWeb Disallow	6	0,07
Youdao Bot Disallow	4	0,05
Diffbot Disallow	1	0,01