Robots.txt Nutzungsverteilung In Tonga

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Googlebot Disallow	2.017	14,5
Baidu Bot Disallow	1.943	13,97
Google Media Partners Disallow	1.533 ^*	11,02
Some Robots Disallow	1.392	10,01
Sogou Bot Disallow	1.048	7,53
Alexa Bot Disallow	907	6,52
Microsoft BingBot Disallow	903	6,49
Yandex Bot Disallow	884	6,35
Yahoo Slurp Disallow	862	6,2
Soso Bot Disallow	856	6,15
Robots Disallow	436	3,13
Ahrefs Bot Disallow	129	0,93
Google AdsBot Disallow	113	0,81
Semrush Bot Disallow	109	0,78
Majestic Bot Disallow	99	0,71
GPTBot Disallow	81	0,58
Google AdsBot Mobile Disallow	78	0,56
Common Crawl Bot Disallow	60	0,43
Google Bard Disallow	55	0,4
AppleBot Disallow	48	0,35
Facebook Bot Disallow	46	0,33
Anthropic Claude Bot Disallow	45	0,32
Apple AI Bot Disallow	40	0,29
Perplexity	31	0,22
Google AdsBot Mobile Apps Disallow	30	0,22
Cohere AI Disallow	29	0,21
Webz Omgili Disallow	20	0,14
Exalead Bot Disallow	15	0,11
ByteDance Bytespider Disallow	14	0,1
Microsoft MSNBot Disallow	14	0,1
Google Image Bot Disallow	14	0,1
Google News Bot Disallow	8	0,06
Diffbot Disallow	7	0,05
DuckDuckGo Bot Disallow	7	0,05
360 Bot Disallow	6	0,04
Google API Bot Disallow	6	0,04
Google Video Bot Disallow	5	0,04
Google Favicon Disallow	5	0,04
YOU Disallow	4	0,03
Google DuplexWeb Disallow	4	0,03
Pipl Disallow	4	0,03
BuiltWithBot Disallow	3	0,02
Youdao Bot Disallow	1	0,01
Google FeedFetcher Disallow	1	0,01