Robots.txt Usage Distribution in Kuwait

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	1,114	19.67
Google AdsBot Mobile Disallow	608	10.74
Googlebot Disallow	540	9.54
Ahrefs Bot Disallow	534	9.43
Robots Disallow	263	4.64
Common Crawl Bot Disallow	202	3.57
GPTBot Disallow	195	3.44
AppleBot Disallow	190	3.36
Google Bard Disallow	190	3.36
Anthropic Claude Bot Disallow	186	3.28
Facebook Bot Disallow	186	3.28
Perplexity	183	3.23
Apple AI Bot Disallow	181	3.2
Cohere AI Disallow	179	3.16
Google AdsBot Mobile Apps Disallow	179	3.16
Baidu Bot Disallow	110	1.94
Some Robots Disallow	78	1.38
Yandex Bot Disallow	72	1.27
Microsoft BingBot Disallow	56	0.99
Yahoo Slurp Disallow	56	0.99
Sogou Bot Disallow	54	0.95
Alexa Bot Disallow	53	0.94
Soso Bot Disallow	50	0.88
Majestic Bot Disallow	46	0.81
Semrush Bot Disallow	36	0.64
Google Media Partners Disallow	30 ^*	0.53
Exalead Bot Disallow	17	0.3
Microsoft MSNBot Disallow	15	0.26
Google Image Bot Disallow	14	0.25
360 Bot Disallow	11	0.19
BuiltWithBot Disallow	11	0.19
Webz Omgili Disallow	7	0.12
ByteDance Bytespider Disallow	6	0.11
DuckDuckGo Bot Disallow	5	0.09
Diffbot Disallow	2	0.04
Google News Bot Disallow	2	0.04
Google Video Bot Disallow	2	0.04