Robots.txt Distribución de uso en Líbano

Distribución para sitios web que utilizan Robots.txt tecnologías

Tecnología	Sitios web	%
Google AdsBot Disallow	1.270	14,03
Ahrefs Bot Disallow	701	7,74
Google AdsBot Mobile Disallow	599	6,62
Common Crawl Bot Disallow	476	5,26
GPTBot Disallow	473	5,23
AppleBot Disallow	420	4,64
Anthropic Claude Bot Disallow	415	4,58
Google Bard Disallow	414	4,57
Facebook Bot Disallow	414	4,57
Apple AI Bot Disallow	410	4,53
Perplexity	408	4,51
Google AdsBot Mobile Apps Disallow	408	4,51
Cohere AI Disallow	407	4,5
Googlebot Disallow	369	4,08
Robots Disallow	296	3,27
Some Robots Disallow	176	1,94
Baidu Bot Disallow	156	1,72
Sogou Bot Disallow	118	1,3
Yandex Bot Disallow	111	1,23
Semrush Bot Disallow	111	1,23
Soso Bot Disallow	99	1,09
Majestic Bot Disallow	99	1,09
Microsoft BingBot Disallow	94	1,04
Yahoo Slurp Disallow	91	1,01
Alexa Bot Disallow	78	0,86
Exalead Bot Disallow	63	0,7
ByteDance Bytespider Disallow	61	0,67
360 Bot Disallow	58	0,64
BuiltWithBot Disallow	55	0,61
Microsoft MSNBot Disallow	50	0,55
Google Media Partners Disallow	45 ^*	0,5
Google Image Bot Disallow	39	0,43
Google News Bot Disallow	28	0,31
Google Video Bot Disallow	27	0,3
Webz Omgili Disallow	6	0,07
Youdao Bot Disallow	5	0,06
YOU Disallow	1	0,01
Pipl Disallow	1	0,01