Robots.txt Usage Distribution in Ukraine

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Yandex Bot Disallow	28,859	29.08
Googlebot Disallow	13,588	13.69
Robots Disallow	12,330	12.42
Google AdsBot Disallow	5,070	5.11
Google AdsBot Mobile Disallow	4,271	4.3
Ahrefs Bot Disallow	3,339	3.36
Baidu Bot Disallow	2,718	2.74
Majestic Bot Disallow	2,646	2.67
Alexa Bot Disallow	2,635	2.65
Microsoft BingBot Disallow	2,522	2.54
Yahoo Slurp Disallow	2,282	2.3
Sogou Bot Disallow	2,171	2.19
Google Media Partners Disallow	2,129 ^*	2.15
Semrush Bot Disallow	1,885	1.9
Some Robots Disallow	1,752	1.77
Soso Bot Disallow	1,721	1.73
Common Crawl Bot Disallow	1,045	1.05
GPTBot Disallow	1,041	1.05
Microsoft MSNBot Disallow	917	0.92
Exalead Bot Disallow	710	0.72
Google Image Bot Disallow	585	0.59
Anthropic Claude Bot Disallow	534	0.54
360 Bot Disallow	518	0.52
Google Bard Disallow	464	0.47
AppleBot Disallow	433	0.44
Facebook Bot Disallow	374	0.38
Apple AI Bot Disallow	323	0.33
Youdao Bot Disallow	315	0.32
Cohere AI Disallow	296	0.3
Perplexity	282	0.28
Google AdsBot Mobile Apps Disallow	260	0.26
ByteDance Bytespider Disallow	251	0.25
Webz Omgili Disallow	194	0.2
YOU Disallow	189	0.19
BuiltWithBot Disallow	137	0.14
DuckDuckGo Bot Disallow	92	0.09
Diffbot Disallow	80	0.08
Google News Bot Disallow	77	0.08
Google Video Bot Disallow	62	0.06
Google API Bot Disallow	49	0.05
Google Favicon Disallow	44	0.04
Google DuplexWeb Disallow	44	0.04
Pipl Disallow	16	0.02
Google FeedFetcher Disallow	2	0
Google Read Aloud Disallow	1	0