Robots.txt Nutzungsverteilung In .tv

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Googlebot Disallow	15.793	15,17
Baidu Bot Disallow	13.190	12,67
Some Robots Disallow	7.482	7,19
Sogou Bot Disallow	7.093	6,81
Yandex Bot Disallow	6.279	6,03
Microsoft BingBot Disallow	6.274	6,03
Alexa Bot Disallow	6.026	5,79
Yahoo Slurp Disallow	5.948	5,71
Soso Bot Disallow	5.692	5,47
Google AdsBot Disallow	3.969	3,81
Google AdsBot Mobile Disallow	3.718	3,57
Robots Disallow	2.982	2,86
Google Media Partners Disallow	2.620 ^*	2,52
Common Crawl Bot Disallow	1.370	1,32
Anthropic Claude Bot Disallow	1.313	1,26
GPTBot Disallow	1.305	1,25
Google Bard Disallow	1.283	1,23
Facebook Bot Disallow	1.267	1,22
AppleBot Disallow	1.225	1,18
Majestic Bot Disallow	1.199	1,15
Apple AI Bot Disallow	1.180	1,13
Cohere AI Disallow	1.078	1,04
Perplexity	1.077	1,03
Google AdsBot Mobile Apps Disallow	1.053	1,01
Ahrefs Bot Disallow	1.036	1
Semrush Bot Disallow	835	0,8
360 Bot Disallow	367	0,35
Microsoft MSNBot Disallow	332	0,32
Webz Omgili Disallow	244	0,23
ByteDance Bytespider Disallow	174	0,17
Exalead Bot Disallow	157	0,15
Google Image Bot Disallow	114	0,11
Diffbot Disallow	79	0,08
BuiltWithBot Disallow	65	0,06
Google News Bot Disallow	47	0,05
Google Video Bot Disallow	46	0,04
DuckDuckGo Bot Disallow	46	0,04
Google Favicon Disallow	44	0,04
Google API Bot Disallow	32	0,03
Google DuplexWeb Disallow	32	0,03
Youdao Bot Disallow	31	0,03
YOU Disallow	15	0,01
Pipl Disallow	7	0,01