Robots.txt Nutzungsverteilung In Sri Lanka

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Google AdsBot Disallow	1.125	13,43
Googlebot Disallow	1.082	12,92
Robots Disallow	955	11,4
Google AdsBot Mobile Disallow	727	8,68
Ahrefs Bot Disallow	482	5,75
Baidu Bot Disallow	326	3,89
Some Robots Disallow	282	3,37
Google Media Partners Disallow	254 ^*	3,03
Yandex Bot Disallow	231	2,76
Alexa Bot Disallow	210	2,51
Sogou Bot Disallow	203	2,42
Microsoft BingBot Disallow	195	2,33
GPTBot Disallow	187	2,23
Yahoo Slurp Disallow	182	2,17
Common Crawl Bot Disallow	176	2,1
Google Bard Disallow	169	2,02
AppleBot Disallow	167	1,99
Anthropic Claude Bot Disallow	165	1,97
Soso Bot Disallow	164	1,96
Facebook Bot Disallow	162	1,93
Apple AI Bot Disallow	161	1,92
Perplexity	157	1,87
Google AdsBot Mobile Apps Disallow	156	1,86
Cohere AI Disallow	155	1,85
Semrush Bot Disallow	91	1,09
Majestic Bot Disallow	74	0,88
Google Image Bot Disallow	38	0,45
Microsoft MSNBot Disallow	19	0,23
Exalead Bot Disallow	13	0,16
ByteDance Bytespider Disallow	12	0,14
Webz Omgili Disallow	10	0,12
DuckDuckGo Bot Disallow	6	0,07
Google News Bot Disallow	5	0,06
Youdao Bot Disallow	5	0,06
360 Bot Disallow	4	0,05
Diffbot Disallow	4	0,05
Google Video Bot Disallow	4	0,05
YOU Disallow	4	0,05
Google Favicon Disallow	4	0,05
Google API Bot Disallow	4	0,05
Google DuplexWeb Disallow	4	0,05
BuiltWithBot Disallow	2	0,02