로봇.txt 사용량 분포 ~에 영국

핀을 꼽는 로봇.txt 사용량 분포 ~에 영국

기술	웹사이트	%
Google AdsBot Disallow	590,308	12.23
Googlebot Disallow	541,245	11.21
Google AdsBot Mobile Disallow	459,740	9.52
Baidu Bot Disallow	236,810	4.9
GPTBot Disallow	181,454	3.76
Common Crawl Bot Disallow	181,119	3.75
Anthropic Claude Bot Disallow	175,954	3.64
Google Bard Disallow	174,435	3.61
AppleBot Disallow	170,127	3.52
Sogou Bot Disallow	170,035	3.52
Facebook Bot Disallow	168,028	3.48
Apple AI Bot Disallow	166,295	3.44
Some Robots Disallow	164,776	3.41
Perplexity	163,762	3.39
Cohere AI Disallow	162,266	3.36
Ahrefs Bot Disallow	157,615	3.26
Google AdsBot Mobile Apps Disallow	155,015	3.21
Yandex Bot Disallow	109,699	2.27
Robots Disallow	108,259	2.24
Microsoft BingBot Disallow	102,180	2.12
Alexa Bot Disallow	101,353	2.1
Yahoo Slurp Disallow	98,303	2.04
Soso Bot Disallow	91,591	1.9
Majestic Bot Disallow	50,583	1.05
Semrush Bot Disallow	45,561	0.94
ByteDance Bytespider Disallow	17,569	0.36
Webz Omgili Disallow	14,355	0.3
Google Media Partners Disallow	11,008 *	0.23
Microsoft MSNBot Disallow	9,909	0.21
Exalead Bot Disallow	8,484	0.18
Diffbot Disallow	8,266	0.17
Google Image Bot Disallow	8,102	0.17
YOU Disallow	6,873	0.14
360 Bot Disallow	3,294	0.07
Youdao Bot Disallow	2,852	0.06
Google News Bot Disallow	2,305	0.05
Google Video Bot Disallow	1,840	0.04
Google Favicon Disallow	1,639	0.03
DuckDuckGo Bot Disallow	1,608	0.03
BuiltWithBot Disallow	1,590	0.03
Google API Bot Disallow	933	0.02
Google DuplexWeb Disallow	844	0.02
Pipl Disallow	378	0.01
Google FeedFetcher Disallow	45	0
Google Read Aloud Disallow	4	0
Google Web Light Disallow	1	0
LLMS Text	1	0
Google BIG Bench Disallow	1	0

4,828,414 탐지

~의 로봇.txt ~에 영국. 마지막 업데이트 04th 1 2025.

Google AdsBot Disallow 현재 이 카테고리에서 가장 인기 있는 기술입니다.