Robots.txt Usage Distribution in Bosnia and Herzegovina

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Robots Disallow	318	12.84
Googlebot Disallow	211	8.52
Google AdsBot Disallow	176	7.11
Ahrefs Bot Disallow	133	5.37
Sogou Bot Disallow	118	4.77
Google AdsBot Mobile Disallow	114	4.6
Soso Bot Disallow	98	3.96
Baidu Bot Disallow	94	3.8
GPTBot Disallow	87	15.1
Yandex Bot Disallow	81	3.27
Common Crawl Bot Disallow	77	13.37
Alexa Bot Disallow	73	12.67
Yahoo Slurp Disallow	73	2.95
Majestic Bot Disallow	70	2.83
Microsoft BingBot Disallow	68	2.75
Semrush Bot Disallow	64	2.58
Some Robots Disallow	51	2.06
Google Bard Disallow	47	8.16
ByteDance Bytespider Disallow	46	7.99
Anthropic Claude Bot Disallow	45	7.81
Facebook Bot Disallow	44	7.64
AppleBot Disallow	43	1.74
Perplexity	42	7.29
Exalead Bot Disallow	42	1.7
360 Bot Disallow	38	1.53
Cohere AI Disallow	36	6.25
Apple AI Bot Disallow	35	6.08
BuiltWithBot Disallow	35	1.41
Google AdsBot Mobile Apps Disallow	34	1.37
Google Media Partners Disallow	13 ^*	0.53
Microsoft MSNBot Disallow	13	0.53
Google Image Bot Disallow	12	0.48
Webz Omgili Disallow	11	1.91
Diffbot Disallow	11	1.91
YOU Disallow	9	1.56
Google News Bot Disallow	2	0.08
Google Video Bot Disallow	2	0.08
DuckDuckGo Bot Disallow	2	0.08
Google Favicon Disallow	2	0.08
Google API Bot Disallow	2	0.08
Google DuplexWeb Disallow	2	0.08
Youdao Bot Disallow	1	0.04
Google FeedFetcher Disallow	1	0.04