Robots.txt Nutzungsverteilung In Tunesien

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Robots Disallow	474	12,26
Googlebot Disallow	348	9
Google AdsBot Disallow	283	7,32
Ahrefs Bot Disallow	252	6,52
Yandex Bot Disallow	201	5,2
Baidu Bot Disallow	184	4,76
Sogou Bot Disallow	178	4,61
Microsoft BingBot Disallow	173	4,48
Soso Bot Disallow	165	4,27
Alexa Bot Disallow	142	3,67
Yahoo Slurp Disallow	134	3,47
Google AdsBot Mobile Disallow	132	3,42
Majestic Bot Disallow	104	2,69
Some Robots Disallow	90	2,33
Semrush Bot Disallow	87	2,25
GPTBot Disallow	83	2,15
Common Crawl Bot Disallow	81	2,1
Anthropic Claude Bot Disallow	71	1,84
Facebook Bot Disallow	67	1,73
Cohere AI Disallow	66	1,71
Exalead Bot Disallow	58	1,5
Microsoft MSNBot Disallow	57	1,47
Google Media Partners Disallow	51 ^*	1,32
ByteDance Bytespider Disallow	44	1,14
Webz Omgili Disallow	43	1,11
360 Bot Disallow	43	1,11
BuiltWithBot Disallow	40	1,03
Google AdsBot Mobile Apps Disallow	33	0,85
Google Bard Disallow	30	0,78
AppleBot Disallow	29	0,75
Perplexity	28	0,72
Apple AI Bot Disallow	27	0,7
Google Image Bot Disallow	19	0,49
Google News Bot Disallow	10	0,26
Google Video Bot Disallow	10	0,26
Google Favicon Disallow	7	0,18
DuckDuckGo Bot Disallow	6	0,16
Google API Bot Disallow	6	0,16
Google DuplexWeb Disallow	6	0,16
Diffbot Disallow	1	0,03
YOU Disallow	1	0,03
Youdao Bot Disallow	1	0,03