Robots.txt Usage Distribution in Australia

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	368,112	12.09
Google AdsBot Mobile Disallow	264,032	8.67
Googlebot Disallow	239,571	7.87
Baidu Bot Disallow	162,809	5.35
GPTBot Disallow	127,926	4.2
Common Crawl Bot Disallow	127,636	4.19
AppleBot Disallow	125,704	4.13
Anthropic Claude Bot Disallow	125,363	4.12
Google Bard Disallow	125,218	4.11
Facebook Bot Disallow	124,724	4.1
Apple AI Bot Disallow	123,644	4.06
Cohere AI Disallow	122,962	4.04
Google AdsBot Mobile Apps Disallow	122,840	4.03
Perplexity	122,834	4.03
Microsoft BingBot Disallow	115,732	3.8
Ahrefs Bot Disallow	105,032	3.45
Sogou Bot Disallow	82,773	2.72
Yandex Bot Disallow	81,054	2.66
Alexa Bot Disallow	79,660	2.62
Yahoo Slurp Disallow	78,004	2.56
Soso Bot Disallow	75,395	2.48
Some Robots Disallow	51,846	1.7
Robots Disallow	40,751	1.34
Majestic Bot Disallow	13,180	0.43
Semrush Bot Disallow	10,543	0.35
Microsoft MSNBot Disallow	4,206	0.14
ByteDance Bytespider Disallow	3,893	0.13
Google Media Partners Disallow	3,840 ^*	0.13
Exalead Bot Disallow	3,504	0.12
Webz Omgili Disallow	3,031	0.1
Google Image Bot Disallow	2,415	0.08
360 Bot Disallow	1,471	0.05
Diffbot Disallow	1,117	0.04
Google News Bot Disallow	642	0.02
Pipl Disallow	583	0.02
Google Video Bot Disallow	547	0.02
DuckDuckGo Bot Disallow	528	0.02
BuiltWithBot Disallow	462	0.02
Google Favicon Disallow	447	0.01
Google API Bot Disallow	254	0.01
YOU Disallow	249	0.01
Google DuplexWeb Disallow	244	0.01
Youdao Bot Disallow	158	0.01
Google FeedFetcher Disallow	14	0
Google Read Aloud Disallow	1	0
Google Web Light Disallow	1	0