Robots.txt Usage Distribution in Tunisia

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Robots Disallow	385	10.81
Googlebot Disallow	339	9.51
Google AdsBot Disallow	260	7.3
Ahrefs Bot Disallow	228	6.4
Yandex Bot Disallow	187	5.25
Baidu Bot Disallow	183	5.14
Sogou Bot Disallow	164	4.6
Microsoft BingBot Disallow	163	4.57
Soso Bot Disallow	157	4.41
Alexa Bot Disallow	136	3.82
Yahoo Slurp Disallow	127	3.56
Google AdsBot Mobile Disallow	122	3.42
Majestic Bot Disallow	97	2.72
Some Robots Disallow	92	2.58
GPTBot Disallow	81	2.27
Semrush Bot Disallow	79	2.22
Common Crawl Bot Disallow	77	2.16
Anthropic Claude Bot Disallow	69	1.94
Facebook Bot Disallow	64	1.8
Cohere AI Disallow	63	1.77
Exalead Bot Disallow	55	1.54
Microsoft MSNBot Disallow	52	1.46
ByteDance Bytespider Disallow	42	1.18
Webz Omgili Disallow	41	1.15
360 Bot Disallow	40	1.12
BuiltWithBot Disallow	37	1.04
Google Bard Disallow	30	0.84
Google Media Partners Disallow	30 ^*	0.84
Google AdsBot Mobile Apps Disallow	29	0.81
Perplexity	28	0.79
AppleBot Disallow	27	0.76
Apple AI Bot Disallow	26	0.73
Google Image Bot Disallow	17	0.48
Google News Bot Disallow	8	0.22
Google Video Bot Disallow	8	0.22
Google Favicon Disallow	5	0.14
DuckDuckGo Bot Disallow	4	0.11
Google API Bot Disallow	4	0.11
Google DuplexWeb Disallow	4	0.11
Diffbot Disallow	1	0.03
Youdao Bot Disallow	1	0.03
YOU Disallow	1	0.03