Robots.txt Usage Distribution in Norfolk Island

Top In Robots.txt Usage Distribution in Norfolk Island

Technology	Websites	%
Googlebot Disallow	10	12.5
Robots Disallow	9	11.25
Yandex Bot Disallow	4	5
Microsoft BingBot Disallow	4	5
Yahoo Slurp Disallow	4	5
Sogou Bot Disallow	3	3.75
Alexa Bot Disallow	3	3.75
Soso Bot Disallow	3	3.75
GPTBot Disallow	3	3.75
Common Crawl Bot Disallow	3	3.75
AppleBot Disallow	3	3.75
Anthropic Claude Bot Disallow	3	3.75
Google Bard Disallow	3	3.75
Facebook Bot Disallow	3	3.75
Apple AI Bot Disallow	3	3.75
Google AdsBot Mobile Apps Disallow	3	3.75
Ahrefs Bot Disallow	2	2.5
Cohere AI Disallow	2	2.5
Perplexity	2	2.5
Semrush Bot Disallow	1	1.25
Microsoft MSNBot Disallow	1	1.25
Webz Omgili Disallow	1	1.25
Google Image Bot Disallow	1	1.25
Google News Bot Disallow	1	1.25
Google Video Bot Disallow	1	1.25
DuckDuckGo Bot Disallow	1	1.25
Google Favicon Disallow	1	1.25
Google API Bot Disallow	1	1.25
Google DuplexWeb Disallow	1	1.25

80 Detections

of Robots.txt in Norfolk Island. Last updated 18th Dec 2024.

Googlebot Disallow is currently the most popular technology in this category.