Robots.txt Usage Distribution in Turkey

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	41,499	21.92
Google AdsBot Disallow	37,491	19.81
Google AdsBot Mobile Disallow	31,173	16.47
Ahrefs Bot Disallow	9,765	5.16
Robots Disallow	9,117	4.82
Baidu Bot Disallow	6,607	3.49
Alexa Bot Disallow	6,158	40.24
Sogou Bot Disallow	5,462	2.89
Majestic Bot Disallow	4,120	2.18
Soso Bot Disallow	3,986	2.11
Semrush Bot Disallow	3,669	1.94
Yandex Bot Disallow	3,593	1.9
Microsoft BingBot Disallow	3,022	1.6
Some Robots Disallow	2,919	1.54
Yahoo Slurp Disallow	2,897	1.53
Google Media Partners Disallow	2,534 ^*	1.34
Exalead Bot Disallow	1,727	0.91
GPTBot Disallow	1,678	10.96
Google Image Bot Disallow	1,561	0.82
360 Bot Disallow	1,411	0.75
Common Crawl Bot Disallow	1,171	7.65
Anthropic Claude Bot Disallow	1,041	6.8
Google Bard Disallow	809	5.29
AppleBot Disallow	791	0.42
Facebook Bot Disallow	721	4.71
Apple AI Bot Disallow	659	4.31
Cohere AI Disallow	645	4.21
Perplexity	635	4.15
Google AdsBot Mobile Apps Disallow	584	0.31
ByteDance Bytespider Disallow	475	3.1
Microsoft MSNBot Disallow	292	0.15
Webz Omgili Disallow	222	1.45
BuiltWithBot Disallow	202	0.11
Diffbot Disallow	130	0.85
DuckDuckGo Bot Disallow	103	0.05
YOU Disallow	73	0.48
Google Video Bot Disallow	63	0.03
Google News Bot Disallow	57	0.03
Youdao Bot Disallow	56	0.03
Google Favicon Disallow	43	0.02
Google API Bot Disallow	43	0.02
Google DuplexWeb Disallow	42	0.02
Pipl Disallow	28	0.01
LLMS Text	12	0.01
Google FeedFetcher Disallow	8	0
Google Read Aloud Disallow	2	0
Google Web Light Disallow	1	0