Robots.txt Usage Distribution in Hungary

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Majestic Bot Disallow	12,041	10.43
Googlebot Disallow	9,712	8.41
Ahrefs Bot Disallow	9,499	8.23
Google AdsBot Disallow	9,477	8.21
Robots Disallow	8,669	7.51
Google AdsBot Mobile Disallow	6,862	5.95
Semrush Bot Disallow	6,125	5.31
Yandex Bot Disallow	4,443	3.85
Baidu Bot Disallow	4,232	3.67
GPTBot Disallow	3,706	3.21
AppleBot Disallow	3,230	2.8
Common Crawl Bot Disallow	3,169	2.75
Anthropic Claude Bot Disallow	3,019	2.62
Alexa Bot Disallow	2,874	2.49
Microsoft BingBot Disallow	2,329	2.02
Google Bard Disallow	2,048	1.77
Facebook Bot Disallow	1,986	1.72
Some Robots Disallow	1,897	1.64
Apple AI Bot Disallow	1,885	1.63
ByteDance Bytespider Disallow	1,864	1.61
Perplexity	1,819	1.58
Cohere AI Disallow	1,795	1.56
Exalead Bot Disallow	1,592	1.38
Sogou Bot Disallow	1,390	1.2
Yahoo Slurp Disallow	1,231	1.07
Soso Bot Disallow	1,156	1
Webz Omgili Disallow	1,121	0.97
Google AdsBot Mobile Apps Disallow	1,003	0.87
Diffbot Disallow	894	0.77
Google Media Partners Disallow	863 ^*	0.75
YOU Disallow	863	0.75
Google Image Bot Disallow	657	0.57
Microsoft MSNBot Disallow	521	0.45
360 Bot Disallow	288	0.25
Google News Bot Disallow	234	0.2
BuiltWithBot Disallow	224	0.19
Google Video Bot Disallow	210	0.18
DuckDuckGo Bot Disallow	142	0.12
Google Favicon Disallow	108	0.09
Google API Bot Disallow	103	0.09
Google DuplexWeb Disallow	102	0.09
Google FeedFetcher Disallow	19	0.02
Pipl Disallow	16	0.01
Youdao Bot Disallow	4	0