Log In
·
Signup for Free
EN
DE
FR
ES
IT
Pусский
한국어
日本語
简体中文
عربي
Tools
Web Technology Trends
Keyword Lists
eCommerce Product Finder
eCommerce Lists
eCommerce Product Lists
Top Sites
Technology Investor
LeadsDiscovery
LeadsEye™
API Access
Datasets
Browser Extensions
CRM Integrations
Features
Lead Generation
Market Analysis
Sales Intelligence
Future Customers
Global Data Coverage
Cyber Risk Auditing
Alternative Financial Data
Report Filtering
Use Cases
All Features
Screencast Demo
Plans
Customers
Resources
Screencast Demo
Knowledge Base
Exportable Fields
FAQ
About Us
Contact Us
Affiliates
Blog
Lookup
Home
Trends
Robots.txt
Robots.txt Usage Distribution unknown
Distribution for websites using Robots.txt technologies
Explain Chart
Top In Robots.txt Usage Distribution unknown
Technology
Websites
%
0 Detections
of Robots.txt unknown. Last updated 07th Mar 2025.
Top 1m
Top 1,000,000 sites by traffic
923,374
Top 100k
Top 100,000 sites by traffic
170,370
Top 10k
Top 10,000 sites by traffic
25,671
Entire Internet
177,919,343
United States
42,128,593
Germany
7,124,348
United Kingdom
4,267,474
Russia
3,680,714
Australia
2,711,102
Canada
1,905,573
Italy
1,470,511
France
1,309,650
Brazil
1,239,422
Netherlands
961,291
.co
945,877
Spain
903,860
Switzerland
850,271
Japan
834,627
India
793,674
China
793,330
Poland
491,752
Belgium
433,399
Sweden
412,991
New Zealand
399,113
.me
376,073
Austria
364,565
Norway
302,583
Czech Republic
300,127
Cocos Islands
298,992
Mexico
289,559
Denmark
280,367
.io
265,037
South Africa
258,802
South Korea
247,298
Ireland
235,749
Turkey
227,950
Romania
226,764
Indonesia
188,050
Finland
180,428
Philippines
165,593
Portugal
144,453
Hungary
132,125
Taiwan
129,905
Singapore
120,129
Argentina
119,472
.tv
107,380
Chile
106,745
Greece
101,914
Anguilla
101,043
Hong Kong
96,272
Ukraine
92,938
Thailand
89,967
Vietnam
88,249
Israel
81,331
Slovakia
80,145
UAE
73,870
Malaysia
65,528
British Virgin Islands
64,812
Luxembourg
58,865
Pakistan
41,358
Samoa
40,749
Saudi Arabia
39,927
Lithuania
38,527
Bulgaria
34,547
Iran
34,044
Slovenia
33,029
Estonia
32,938
Latvia
31,955
Cameroon
31,699
Iceland
31,660
Peru
30,894
Croatia
21,951
Isle of Man
21,209
Niue
20,185
Puerto Rico
19,217
Serbia
18,903
Costa Rica
18,779
Palau
18,173
Ecuador
17,700
Nigeria
17,524
Belarus
16,900
Kenya
16,403
Reunion
15,557
Uruguay
15,462
Liechtenstein
14,968
Tonga
14,590
Kazakhstan
13,212
Guernsey
13,143
Saint Vincent and the Grenadines
13,100
Bangladesh
12,727
Saint Helena
12,655
Dominican Republic
12,550
Morocco
11,318
Malta
11,144
Egypt
10,412
Guatemala
9,822
Saint Pierre and Miquelon
8,244
Belize
8,103
Panama
7,975
Albania
7,837
Lebanon
7,813
Georgia
7,797
Sri Lanka
7,576
Armenia
7,531
Venezuela
7,482
Cyprus
7,452
Somalia
7,374
Uganda
6,704
Qatar
6,560
Micronesia
6,428
Ghana
6,237
Uzbekistan
5,960
Tanzania
5,468
Christmas Island
5,251
Kuwait
5,229
Nepal
5,037
Macedonia
4,590
Moldova
4,527
Jordan
4,437
Honduras
4,179
Paraguay
3,791
Mauritius
3,758
Tunisia
3,432
Guadeloupe
3,422
Mongolia
3,373
Ethiopia
3,243
Antigua and Barbuda
3,214
Azerbaijan
3,208
El Salvador
3,100
Bahrain
3,097
Cambodia
3,027
Bahamas
2,976
Algeria
2,944
Bolivia
2,934
Oman
2,841
Monaco
2,831
Bosnia and Herzegovina
2,751
Sao Tome and Principe
2,653
Namibia
2,472
Rwanda
2,463
Jersey
2,325
Cayman Islands
2,323
United States Virgin Islands
2,299
Zimbabwe
2,290
Faroe Islands
2,204
Trinidad and Tobago
2,081
Afghanistan
2,059
Zambia
2,045
Seychelles
1,994
Jamaica
1,958
Sint Maarten
1,892
Martinique
1,871
Bermuda
1,764
Cote d'Ivoire
1,708
Montserrat
1,677
Saint Lucia
1,636
Mayotte
1,587
Iraq
1,564
Maldives
1,558
Andorra
1,545
New Caledonia
1,529
Myanmar
1,527
Mozambique
1,502
Nicaragua
1,499
American Samoa
1,451
Barbados
1,409
Angola
1,394
Kyrgyzstan
1,269
Grenada
1,266
Gibraltar
1,241
Senegal
1,200
Fiji
1,151
Botswana
1,142
Congo
1,121
Greenland
1,095
Turks and Caicos Islands
1,093
Guyana
1,088
Madagascar
1,005
Aruba
926
South Georgia and the South Sandwich Islands
907
Haiti
856
Palestine
820
Burkina Faso
706
Wallis and Futuna
678
Niger
674
Togo
660
French Guiana
613
Vanuatu
591
Gambia
582
Cuba
572
Tajikistan
556
Mali
546
Malawi
532
Djibouti
458
Benin
446
Brunei
394
Solomon Islands
360
Sierra Leone
358
Papua New Guinea
357
Åland Islands
351
Suriname
311
Syria
303
Burundi
295
Sudan
295
Guam
294
Yemen
293
East Timor
287
Lao People's Democratic Republic
273
Liberia
239
French Polynesia
238
Heard Island and McDonald Islands
223
Turkmenistan
222
Libya
217
San Marino
183
Swaziland
163
Lesotho
159
Curacao
159
Bhutan
157
Mauritania
156
Saint Kitts and Nevis
150
Cook Islands
127
Gabon
125
Svalbard and Jan Mayen
114
Macao
111
Northern Mariana Islands
110
Dominica
101
Guinea
82
United States Minor Outlying Islands
81
Norfolk Island
77
Congo
64
South Sudan
54
Chad
53
Cabo Verde
47
Falkland Islands
40
Pitcairn
39
Kiribati
33
Saint Barthélemy
30
Marshall Islands
30
Guinea-Bissau
28
Antarctica
25
Equatorial Guinea
22
Bonaire
20
Nauru
18
Vatican City
18
Robots.txt Categories
All Robots.txt
AI Bot
us
de
gb
ru
au
ca
it
fr
br
nl
DotCo
es
ch
jp
in
cn