Robots.txt Usage Distribution in Turkey

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	53,250	23.31
Google AdsBot Disallow	49,536	21.69
Google AdsBot Mobile Disallow	41,973	18.38
Ahrefs Bot Disallow	10,783	4.72
Robots Disallow	7,380	3.23
Alexa Bot Disallow	6,876	3.01
Baidu Bot Disallow	6,729	2.95
GPTBot Disallow	5,668	2.48
Majestic Bot Disallow	4,967	2.17
Semrush Bot Disallow	4,162	1.82
Sogou Bot Disallow	4,130	1.81
Soso Bot Disallow	3,682	1.61
Yandex Bot Disallow	3,448	1.51
Microsoft BingBot Disallow	2,704	1.18
Some Robots Disallow	2,622	1.15
Yahoo Slurp Disallow	2,578	1.13
Google Media Partners Disallow	2,054 ^*	0.9
Google Image Bot Disallow	1,874	0.82
Exalead Bot Disallow	1,806	0.79
360 Bot Disallow	1,470	0.64
Common Crawl Bot Disallow	1,411	0.62
Anthropic Claude Bot Disallow	1,206	0.53
Google Bard Disallow	1,009	0.44
AppleBot Disallow	922	0.4
Facebook Bot Disallow	879	0.38
Apple AI Bot Disallow	829	0.36
Cohere AI Disallow	809	0.35
Google AdsBot Mobile Apps Disallow	782	0.34
Perplexity	776	0.34
ByteDance Bytespider Disallow	507	0.22
Microsoft MSNBot Disallow	361	0.16
Webz Omgili Disallow	287	0.13
BuiltWithBot Disallow	242	0.11
Diffbot Disallow	92	0.04
DuckDuckGo Bot Disallow	91	0.04
Google Video Bot Disallow	85	0.04
Google News Bot Disallow	80	0.04
Youdao Bot Disallow	73	0.03
Google API Bot Disallow	63	0.03
Google Favicon Disallow	62	0.03
Google DuplexWeb Disallow	61	0.03
YOU Disallow	49	0.02
Pipl Disallow	13	0.01
Google FeedFetcher Disallow	11	0
Google Read Aloud Disallow	1	0
Google Web Light Disallow	1	0