Robots.txt Nutzungsverteilung In Indonesien

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Googlebot Disallow	24.438	12,03
Robots Disallow	19.762	9,73
Baidu Bot Disallow	18.527	9,12
Some Robots Disallow	15.685	7,72
Alexa Bot Disallow	14.419	7,1
Yahoo Slurp Disallow	13.351	6,57
Microsoft MSNBot Disallow	10.299	5,07
Google Image Bot Disallow	9.741	4,79
Yandex Bot Disallow	7.527	3,71
Ahrefs Bot Disallow	6.116	3,01
Sogou Bot Disallow	6.067	2,99
Google AdsBot Disallow	5.955	2,93
Majestic Bot Disallow	5.331	2,62
Microsoft BingBot Disallow	4.991	2,46
Google AdsBot Mobile Disallow	4.416	2,17
Soso Bot Disallow	4.312	2,12
Semrush Bot Disallow	3.895	1,92
GPTBot Disallow	3.423	1,68
Common Crawl Bot Disallow	3.236	1,59
Google Media Partners Disallow	2.925 ^*	1,44
Anthropic Claude Bot Disallow	2.814	1,39
Exalead Bot Disallow	1.659	0,82
Google Bard Disallow	1.559	0,77
Facebook Bot Disallow	1.447	0,71
Perplexity	1.381	0,68
ByteDance Bytespider Disallow	1.346	0,66
AppleBot Disallow	1.337	0,66
Apple AI Bot Disallow	1.178	0,58
Google AdsBot Mobile Apps Disallow	1.080	0,53
Cohere AI Disallow	1.071	0,53
Webz Omgili Disallow	960	0,47
Pipl Disallow	730	0,36
Youdao Bot Disallow	525	0,26
Diffbot Disallow	321	0,16
YOU Disallow	276	0,14
360 Bot Disallow	233	0,11
DuckDuckGo Bot Disallow	171	0,08
BuiltWithBot Disallow	152	0,07
Google News Bot Disallow	131	0,06
Google Video Bot Disallow	107	0,05
Google Favicon Disallow	92	0,05
Google API Bot Disallow	83	0,04
Google DuplexWeb Disallow	83	0,04
Google FeedFetcher Disallow	3	0
Google Web Light Disallow	1	0