Robots.txt Nutzungsverteilung in Weißrussland

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Yandex Bot Disallow	5.269	31,18
Robots Disallow	2.572	15,22
Googlebot Disallow	2.432	14,39
Semrush Bot Disallow	603	3,57
Microsoft BingBot Disallow	515	3,05
Ahrefs Bot Disallow	461	2,73
Majestic Bot Disallow	446	2,64
Some Robots Disallow	445	2,63
Alexa Bot Disallow	442	2,62
Google Media Partners Disallow	351 ^*	2,08
GPTBot Disallow	310	1,83
Common Crawl Bot Disallow	304	1,8
Baidu Bot Disallow	302	1,79
Yahoo Slurp Disallow	286	1,69
Microsoft MSNBot Disallow	265	1,57
Anthropic Claude Bot Disallow	261	1,54
Google Bard Disallow	246	1,46
Sogou Bot Disallow	235	1,39
Webz Omgili Disallow	230	1,36
Soso Bot Disallow	202	1,2
Google AdsBot Disallow	129	0,76
Google AdsBot Mobile Disallow	102	0,6
Exalead Bot Disallow	79	0,47
Google Image Bot Disallow	76	0,45
AppleBot Disallow	42	0,25
Facebook Bot Disallow	34	0,2
Apple AI Bot Disallow	29	0,17
Google AdsBot Mobile Apps Disallow	27	0,16
360 Bot Disallow	23	0,14
Cohere AI Disallow	22	0,13
Perplexity	20	0,12
ByteDance Bytespider Disallow	19	0,11
DuckDuckGo Bot Disallow	19	0,11
Google News Bot Disallow	18	0,11
Google Video Bot Disallow	16	0,09
Google API Bot Disallow	14	0,08
Google Favicon Disallow	13	0,08
Google DuplexWeb Disallow	13	0,08
BuiltWithBot Disallow	9	0,05
Diffbot Disallow	7	0,04
YOU Disallow	5	0,03
Pipl Disallow	4	0,02
Google FeedFetcher Disallow	2	0,01
Youdao Bot Disallow	1	0,01