Robots.txt Nutzungsverteilung In Singapur

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Google AdsBot Disallow	25.768	19
Google AdsBot Mobile Disallow	18.771	13,84
Googlebot Disallow	16.447	12,13
Ahrefs Bot Disallow	7.476	5,51
GPTBot Disallow	4.922	3,63
Common Crawl Bot Disallow	4.835	3,56
Anthropic Claude Bot Disallow	4.824	3,56
AppleBot Disallow	4.782	3,53
Google Bard Disallow	4.772	3,52
Facebook Bot Disallow	4.724	3,48
Apple AI Bot Disallow	4.697	3,46
Perplexity	4.630	3,41
Google AdsBot Mobile Apps Disallow	4.610	3,4
Cohere AI Disallow	4.593	3,39
Robots Disallow	3.852	2,84
Baidu Bot Disallow	2.177	1,6
Some Robots Disallow	1.705	1,26
Majestic Bot Disallow	1.527	1,13
Microsoft BingBot Disallow	1.415	1,04
Yandex Bot Disallow	1.402	1,03
Alexa Bot Disallow	1.400	1,03
Yahoo Slurp Disallow	1.120	0,83
Sogou Bot Disallow	1.073	0,79
Soso Bot Disallow	946	0,7
Semrush Bot Disallow	909	0,67
Google Media Partners Disallow	491 ^*	0,36
Exalead Bot Disallow	408	0,3
Google Image Bot Disallow	297	0,22
Microsoft MSNBot Disallow	262	0,19
Webz Omgili Disallow	186	0,14
ByteDance Bytespider Disallow	159	0,12
Diffbot Disallow	62	0,05
DuckDuckGo Bot Disallow	57	0,04
Google News Bot Disallow	54	0,04
Google Video Bot Disallow	51	0,04
Google Favicon Disallow	47	0,03
360 Bot Disallow	41	0,03
Youdao Bot Disallow	32	0,02
YOU Disallow	28	0,02
Google API Bot Disallow	25	0,02
Google DuplexWeb Disallow	24	0,02
Pipl Disallow	24	0,02
BuiltWithBot Disallow	18	0,01