Robots.txt Distribuzione dell'utilizzo su tutta Internet

Distribuzione per i siti Web che utilizzano Robots.txt tecnologie

Appuntare Robots.txt Distribuzione dell'utilizzo su tutta Internet

Tecnologia	Siti web	%
Some Robots Disallow	39.138.226	20,18
Googlebot Disallow	17.586.304	9,07
Baidu Bot Disallow	13.019.397	6,71
Google Media Partners Disallow	11.836.246	6,1
Google AdsBot Disallow	11.034.816	5,69
Sogou Bot Disallow	9.657.154	4,98
Google AdsBot Mobile Disallow	8.178.514	4,22
Yandex Bot Disallow	6.435.604	3,32
Robots Disallow	6.213.656	3,2
Microsoft BingBot Disallow	6.118.657	3,15
Alexa Bot Disallow	5.820.730	3
Yahoo Slurp Disallow	5.599.101	2,89
Soso Bot Disallow	5.417.211	2,79
Ahrefs Bot Disallow	5.237.782	2,7
GPTBot Disallow	3.718.514	1,92
Common Crawl Bot Disallow	3.714.185	1,91
AppleBot Disallow	3.591.626	1,85
Anthropic Claude Bot Disallow	3.494.987	1,8
Google Bard Disallow	3.338.341	1,72
Facebook Bot Disallow	3.303.694	1,7
Apple AI Bot Disallow	3.153.903	1,63
Cohere AI Disallow	3.125.101	1,61
Perplexity	3.066.202	1,58
Google AdsBot Mobile Apps Disallow	2.927.825	1,51
Majestic Bot Disallow	2.655.466	1,37
Semrush Bot Disallow	2.347.394	1,21
ByteDance Bytespider Disallow	871.069	0,45
Exalead Bot Disallow	553.678	0,29
Microsoft MSNBot Disallow	486.263	0,25
Webz Omgili Disallow	471.042	0,24
360 Bot Disallow	390.370	0,2
Google Image Bot Disallow	376.502	0,19
BuiltWithBot Disallow	198.563	0,1
Diffbot Disallow	195.304	0,1
Google News Bot Disallow	106.128	0,05
Google Video Bot Disallow	104.138	0,05
YOU Disallow	101.803	0,05
DuckDuckGo Bot Disallow	98.360	0,05
Google Favicon Disallow	86.701	0,04
Youdao Bot Disallow	80.635	0,04
Google API Bot Disallow	55.670	0,03
Google DuplexWeb Disallow	54.589	0,03
Pipl Disallow	20.766	0,01
Google FeedFetcher Disallow	3.997	0
Google Read Aloud Disallow	445	0
Google Web Light Disallow	112	0
Google BIG Bench Disallow	33	0