Robots.txt Nutzungsverteilung in Tunesien

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Robots Disallow	513	16,84
Googlebot Disallow	302	9,91
Google AdsBot Disallow	220	7,22
Ahrefs Bot Disallow	183	6,01
Yandex Bot Disallow	172	5,65
Baidu Bot Disallow	171	5,61
Sogou Bot Disallow	158	5,19
Microsoft BingBot Disallow	146	4,79
Soso Bot Disallow	135	4,43
Yahoo Slurp Disallow	130	4,27
Alexa Bot Disallow	128	29,02
Google AdsBot Mobile Disallow	101	3,32
Some Robots Disallow	74	2,43
Majestic Bot Disallow	67	2,2
Semrush Bot Disallow	55	1,81
GPTBot Disallow	46	10,43
Common Crawl Bot Disallow	43	9,75
Anthropic Claude Bot Disallow	37	8,39
Facebook Bot Disallow	34	7,71
Cohere AI Disallow	34	7,71
Microsoft MSNBot Disallow	33	1,08
Exalead Bot Disallow	28	0,92
Google AdsBot Mobile Apps Disallow	22	0,72
ByteDance Bytespider Disallow	22	4,99
Webz Omgili Disallow	20	4,54
360 Bot Disallow	19	0,62
Google Bard Disallow	18	4,08
BuiltWithBot Disallow	17	0,56
AppleBot Disallow	16	0,53
Perplexity	16	3,63
Apple AI Bot Disallow	15	3,4
Google Image Bot Disallow	15	0,49
Google Media Partners Disallow	13 ^*	0,43
Google News Bot Disallow	8	0,26
Google Video Bot Disallow	7	0,23
DuckDuckGo Bot Disallow	7	0,23
Google Favicon Disallow	7	0,23
Google API Bot Disallow	6	0,2
Google DuplexWeb Disallow	6	0,2
Diffbot Disallow	1	0,23
YOU Disallow	1	0,23