Robots.txt Nutzungsverteilung In Indonesien

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Googlebot Disallow	24.219	12,88
Baidu Bot Disallow	18.684	9,93
Some Robots Disallow	15.696	8,35
Robots Disallow	15.239	8,1
Alexa Bot Disallow	12.694	6,75
Yahoo Slurp Disallow	11.687	6,21
Microsoft MSNBot Disallow	9.468	5,03
Google Image Bot Disallow	8.914	4,74
Yandex Bot Disallow	6.970	3,71
Ahrefs Bot Disallow	6.241	3,32
Google AdsBot Disallow	6.186	3,29
Majestic Bot Disallow	4.986	2,65
Sogou Bot Disallow	4.632	2,46
Google AdsBot Mobile Disallow	4.624	2,46
Microsoft BingBot Disallow	4.141	2,2
Semrush Bot Disallow	3.963	2,11
Soso Bot Disallow	3.632	1,93
GPTBot Disallow	3.411	1,81
Common Crawl Bot Disallow	3.051	1,62
Anthropic Claude Bot Disallow	2.765	1,47
Google Media Partners Disallow	2.337 ^*	1,24
Exalead Bot Disallow	1.535	0,82
Google Bard Disallow	1.488	0,79
ByteDance Bytespider Disallow	1.345	0,72
AppleBot Disallow	1.285	0,68
Facebook Bot Disallow	1.198	0,64
Apple AI Bot Disallow	1.159	0,62
Perplexity	1.124	0,6
Google AdsBot Mobile Apps Disallow	1.066	0,57
Cohere AI Disallow	1.033	0,55
Webz Omgili Disallow	890	0,47
Youdao Bot Disallow	688	0,37
Pipl Disallow	675	0,36
360 Bot Disallow	191	0,1
DuckDuckGo Bot Disallow	166	0,09
BuiltWithBot Disallow	117	0,06
Google News Bot Disallow	117	0,06
Google Video Bot Disallow	92	0,05
Google Favicon Disallow	85	0,05
Diffbot Disallow	78	0,04
Google API Bot Disallow	76	0,04
Google DuplexWeb Disallow	76	0,04
YOU Disallow	43	0,02
Google FeedFetcher Disallow	3	0
Google Web Light Disallow	1	0