Robots.txt Nutzungsverteilung in São Tomé und Príncipe

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Robots Disallow	317	12,16
Googlebot Disallow	260	9,97
Baidu Bot Disallow	240	9,21
Sogou Bot Disallow	227	8,71
Microsoft BingBot Disallow	155	5,95
Yandex Bot Disallow	152	5,83
Alexa Bot Disallow	146	28,24
Yahoo Slurp Disallow	144	5,52
Soso Bot Disallow	139	5,33
Google Media Partners Disallow	133 ^*	5,1
Some Robots Disallow	122	4,68
GPTBot Disallow	51	9,86
Google AdsBot Disallow	40	1,53
Common Crawl Bot Disallow	40	7,74
Google Bard Disallow	38	7,35
Anthropic Claude Bot Disallow	37	7,16
Facebook Bot Disallow	34	6,58
AppleBot Disallow	32	1,23
Apple AI Bot Disallow	29	5,61
Webz Omgili Disallow	29	5,61
Ahrefs Bot Disallow	28	1,07
Cohere AI Disallow	22	4,26
Perplexity	21	4,06
Google AdsBot Mobile Disallow	20	0,77
Majestic Bot Disallow	18	0,69
Semrush Bot Disallow	18	0,69
ByteDance Bytespider Disallow	18	3,48
Diffbot Disallow	14	2,71
Google AdsBot Mobile Apps Disallow	12	0,46
YOU Disallow	12	2,32
Microsoft MSNBot Disallow	11	0,42
Google Image Bot Disallow	8	0,31
Google News Bot Disallow	5	0,19
Google Video Bot Disallow	5	0,19
DuckDuckGo Bot Disallow	5	0,19
Pipl Disallow	5	0,19
Google Favicon Disallow	4	0,15
Google API Bot Disallow	4	0,15
Google DuplexWeb Disallow	4	0,15
360 Bot Disallow	3	0,12
Exalead Bot Disallow	2	0,08
BuiltWithBot Disallow	1	0,04
Youdao Bot Disallow	1	0,04
LLMS Text	1	0,04