Robots.txt Usage Distribution in New Caledonia

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	317	19.24
Google AdsBot Disallow	281	17.05
Google AdsBot Mobile Disallow	247	14.99
Robots Disallow	152	9.22
Ahrefs Bot Disallow	77	4.67
Google Image Bot Disallow	67	4.07
Some Robots Disallow	61	3.7
Google Media Partners Disallow	51 ^*	3.09
Semrush Bot Disallow	48	2.91
Majestic Bot Disallow	30	1.82
Webz Omgili Disallow	27	1.64
Sogou Bot Disallow	20	1.21
Microsoft BingBot Disallow	20	1.21
Yahoo Slurp Disallow	20	1.21
DuckDuckGo Bot Disallow	19	1.15
Google AdsBot Mobile Apps Disallow	16	0.97
Baidu Bot Disallow	15	0.91
Yandex Bot Disallow	15	0.91
Alexa Bot Disallow	14	0.85
Common Crawl Bot Disallow	11	0.67
Microsoft MSNBot Disallow	11	0.67
Google News Bot Disallow	11	0.67
Google Video Bot Disallow	11	0.67
Google Favicon Disallow	11	0.67
Google API Bot Disallow	11	0.67
Google DuplexWeb Disallow	11	0.67
GPTBot Disallow	10	0.61
Google Bard Disallow	10	0.61
AppleBot Disallow	9	0.55
Anthropic Claude Bot Disallow	9	0.55
Facebook Bot Disallow	9	0.55
Apple AI Bot Disallow	8	0.49
Perplexity	7	0.42
Cohere AI Disallow	5	0.3
ByteDance Bytespider Disallow	4	0.24
Soso Bot Disallow	2	0.12
Exalead Bot Disallow	1	0.06