Robots.txt Nutzungsverteilung In Kuwait

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Google AdsBot Disallow	1.082	18,37
Google AdsBot Mobile Disallow	574	9,74
Ahrefs Bot Disallow	567	9,62
Googlebot Disallow	527	8,95
Robots Disallow	319	5,42
Common Crawl Bot Disallow	212	3,6
GPTBot Disallow	199	3,38
AppleBot Disallow	199	3,38
Google Bard Disallow	198	3,36
Anthropic Claude Bot Disallow	194	3,29
Facebook Bot Disallow	194	3,29
Perplexity	192	3,26
Apple AI Bot Disallow	190	3,23
Cohere AI Disallow	187	3,17
Google AdsBot Mobile Apps Disallow	185	3,14
Baidu Bot Disallow	112	1,9
Yandex Bot Disallow	90	1,53
Sogou Bot Disallow	85	1,44
Some Robots Disallow	81	1,37
Microsoft BingBot Disallow	73	1,24
Yahoo Slurp Disallow	70	1,19
Alexa Bot Disallow	68	1,15
Soso Bot Disallow	68	1,15
Majestic Bot Disallow	51	0,87
Semrush Bot Disallow	37	0,63
Google Media Partners Disallow	34 ^*	0,58
Exalead Bot Disallow	20	0,34
Google Image Bot Disallow	15	0,25
Microsoft MSNBot Disallow	14	0,24
360 Bot Disallow	13	0,22
BuiltWithBot Disallow	13	0,22
ByteDance Bytespider Disallow	8	0,14
Webz Omgili Disallow	8	0,14
DuckDuckGo Bot Disallow	5	0,08
Diffbot Disallow	3	0,05
Google News Bot Disallow	2	0,03
Google Video Bot Disallow	2	0,03