Robots.txt Usage Distribution in Pakistan

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Ahrefs Bot Disallow	9,905	26.55
Google AdsBot Disallow	9,704	26.01
Googlebot Disallow	2,501	6.7
Robots Disallow	2,501	6.7
Baidu Bot Disallow	1,850	4.96
Sogou Bot Disallow	1,600	4.29
Some Robots Disallow	1,378	3.69
Yandex Bot Disallow	982	2.63
Microsoft BingBot Disallow	920	2.47
Yahoo Slurp Disallow	883	2.37
Alexa Bot Disallow	826	40.21
Soso Bot Disallow	814	2.18
Majestic Bot Disallow	400	1.07
Google AdsBot Mobile Disallow	399	1.07
Semrush Bot Disallow	363	0.97
Google Media Partners Disallow	345 ^*	0.92
Common Crawl Bot Disallow	211	10.27
GPTBot Disallow	187	9.1
Google Image Bot Disallow	183	0.49
AppleBot Disallow	134	0.36
ByteDance Bytespider Disallow	127	6.18
Google Bard Disallow	126	6.13
Anthropic Claude Bot Disallow	111	5.4
Facebook Bot Disallow	104	5.06
Exalead Bot Disallow	101	0.27
Apple AI Bot Disallow	90	4.38
Perplexity	89	4.33
Cohere AI Disallow	87	4.24
Google AdsBot Mobile Apps Disallow	81	0.22
360 Bot Disallow	63	0.17
BuiltWithBot Disallow	62	0.17
Microsoft MSNBot Disallow	50	0.13
Webz Omgili Disallow	22	1.07
Diffbot Disallow	21	1.02
Google Video Bot Disallow	20	0.05
Google News Bot Disallow	17	0.05
Youdao Bot Disallow	14	0.04
DuckDuckGo Bot Disallow	8	0.02
YOU Disallow	6	0.29
Google Favicon Disallow	5	0.01
Google API Bot Disallow	5	0.01
Google DuplexWeb Disallow	5	0.01
Pipl Disallow	3	0.01