Robots.txt Usage Distribution in Algeria

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google Media Partners Disallow	327 ^*	12.17
Robots Disallow	305	11.36
Googlebot Disallow	265	9.87
Google AdsBot Disallow	262	9.75
Ahrefs Bot Disallow	164	6.11
Google AdsBot Mobile Disallow	146	5.44
Baidu Bot Disallow	133	4.95
Yandex Bot Disallow	103	3.83
Yahoo Slurp Disallow	85	3.16
Microsoft BingBot Disallow	79	2.94
Some Robots Disallow	75	2.79
Sogou Bot Disallow	74	2.76
Alexa Bot Disallow	67	2.49
Soso Bot Disallow	66	2.46
Majestic Bot Disallow	52	1.94
Semrush Bot Disallow	45	1.68
GPTBot Disallow	42	1.56
Common Crawl Bot Disallow	36	1.34
Google Bard Disallow	32	1.19
AppleBot Disallow	30	1.12
Microsoft MSNBot Disallow	29	1.08
Anthropic Claude Bot Disallow	28	1.04
Google AdsBot Mobile Apps Disallow	28	1.04
Facebook Bot Disallow	22	0.82
Perplexity	22	0.82
Apple AI Bot Disallow	21	0.78
Cohere AI Disallow	20	0.74
ByteDance Bytespider Disallow	14	0.52
Google Image Bot Disallow	14	0.52
Exalead Bot Disallow	11	0.41
Google News Bot Disallow	10	0.37
Google Video Bot Disallow	10	0.37
DuckDuckGo Bot Disallow	10	0.37
Google Favicon Disallow	10	0.37
Google API Bot Disallow	10	0.37
Google DuplexWeb Disallow	10	0.37
Webz Omgili Disallow	7	0.26
360 Bot Disallow	7	0.26
BuiltWithBot Disallow	4	0.15
YOU Disallow	4	0.15
Diffbot Disallow	3	0.11
Youdao Bot Disallow	3	0.11
Pipl Disallow	1	0.04