Robots.txt Usage Distribution in Qatar

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	1,121	16.34
Googlebot Disallow	814	11.87
Google AdsBot Mobile Disallow	668	9.74
Ahrefs Bot Disallow	483	7.04
Robots Disallow	377	5.5
Baidu Bot Disallow	255	3.72
Yandex Bot Disallow	225	3.28
Alexa Bot Disallow	197	2.87
Yahoo Slurp Disallow	195	2.84
Microsoft BingBot Disallow	190	2.77
Sogou Bot Disallow	189	2.76
Soso Bot Disallow	174	2.54
GPTBot Disallow	167	2.43
Common Crawl Bot Disallow	164	2.39
AppleBot Disallow	155	2.26
Google Bard Disallow	153	2.23
Anthropic Claude Bot Disallow	151	2.2
Facebook Bot Disallow	148	2.16
Perplexity	146	2.13
Apple AI Bot Disallow	145	2.11
Cohere AI Disallow	139	2.03
Google AdsBot Mobile Apps Disallow	139	2.03
Some Robots Disallow	116	1.69
Majestic Bot Disallow	90	1.31
Google Media Partners Disallow	89 ^*	1.3
Semrush Bot Disallow	71	1.04
Microsoft MSNBot Disallow	29	0.42
ByteDance Bytespider Disallow	16	0.23
Exalead Bot Disallow	15	0.22
Webz Omgili Disallow	10	0.15
Google Image Bot Disallow	10	0.15
360 Bot Disallow	4	0.06
BuiltWithBot Disallow	3	0.04
Diffbot Disallow	2	0.03
Google News Bot Disallow	2	0.03
Google Video Bot Disallow	2	0.03
DuckDuckGo Bot Disallow	1	0.01
Google Favicon Disallow	1	0.01
YOU Disallow	1	0.01
Google API Bot Disallow	1	0.01
Pipl Disallow	1	0.01