Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives
Latest crawl: CC-MAIN-2024-10
Just the distribution (percentage / relative frequency) of URLs on top-level domains for few selected crawls, include the latest monthly crawl… Only TLDs which reach at least 0.05% in one of the selected crawls are shown.
crawl | CC-MAIN-2008-2009 | CC-MAIN-2012 | CC-MAIN-2014-10 | CC-MAIN-2016-30 | CC-MAIN-2019-04 | CC-MAIN-2020-40 | CC-MAIN-2024-10 |
---|---|---|---|---|---|---|---|
tld | |||||||
ae | 0.0010 | 0.0102 | 0.0123 | 0.0099 | 0.0239 | 0.0296 | 0.0576 |
app | NaN | NaN | NaN | NaN | 0.0030 | 0.0131 | 0.1756 |
ar | 0.0818 | 0.2381 | 0.0991 | 0.0782 | 0.2117 | 0.2407 | 0.3506 |
at | 0.0025 | 0.3797 | 0.0632 | 0.1556 | 0.3867 | 0.4158 | 0.4779 |
au | 0.6008 | 0.9935 | 0.4579 | 0.5644 | 0.8954 | 1.0714 | 1.0763 |
be | 0.0024 | 0.3972 | 0.0564 | 0.1547 | 0.4185 | 0.5491 | 0.5467 |
bg | 0.0010 | 0.0379 | 0.0078 | 0.0161 | 0.1224 | 0.1391 | 0.1489 |
biz | 0.5549 | 0.3747 | 0.0384 | 0.0532 | 0.2325 | 0.1758 | 0.1468 |
blog | NaN | NaN | NaN | NaN | 0.0610 | 0.1393 | 0.0311 |
br | 0.3366 | 0.7642 | 0.2175 | 0.2958 | 1.1499 | 1.4005 | 1.3012 |
by | 0.0023 | 0.0353 | 0.0009 | 0.0138 | 0.2198 | 0.2512 | 0.1825 |
ca | 0.0141 | 0.6900 | 0.4690 | 0.5206 | 0.7015 | 0.8269 | 0.9839 |
cat | 0.0004 | 0.0496 | 0.0103 | 0.0150 | 0.0837 | 0.1320 | 0.1300 |
cc | 0.0035 | 0.0957 | 0.0404 | 0.1105 | 0.1135 | 0.1257 | 0.1197 |
ch | 0.0033 | 0.5266 | 0.1018 | 0.2150 | 0.5340 | 0.5732 | 0.6266 |
cl | 0.0010 | 0.0834 | 0.0660 | 0.0633 | 0.1249 | 0.1386 | 0.2329 |
club | NaN | NaN | 0.0000 | 0.0004 | 0.1096 | 0.0851 | 0.0694 |
cn | 1.0883 | 0.4208 | 0.0157 | 0.3096 | 0.7695 | 1.4831 | 0.7648 |
co | 0.0165 | 0.0454 | 0.0328 | 0.0638 | 0.4746 | 0.3649 | 0.4554 |
com | 71.2867 | 55.8058 | 74.6955 | 69.1877 | 47.4892 | 45.0745 | 45.1394 |
cz | 0.0065 | 0.7827 | 0.0480 | 0.2279 | 1.0218 | 1.1037 | 1.0053 |
de | 0.0523 | 4.8447 | 0.7005 | 1.5108 | 3.8229 | 4.0912 | 4.1195 |
dk | 0.0033 | 0.3705 | 0.0803 | 0.1525 | 0.3622 | 0.4187 | 0.4274 |
edu | 1.2783 | 0.6182 | 3.0895 | 2.9479 | 0.5654 | 0.9903 | 1.1618 |
ee | 0.0003 | 0.0482 | 0.0058 | 0.0134 | 0.1147 | 0.1481 | 0.1536 |
es | 0.0339 | 0.6739 | 0.1764 | 0.2350 | 0.8740 | 0.9364 | 1.0492 |
eu | 0.0045 | 0.4590 | 0.1181 | 0.1842 | 0.6887 | 0.8290 | 0.7092 |
fi | 0.0009 | 0.1757 | 0.0262 | 0.0514 | 0.2920 | 0.3356 | 0.3719 |
fm | 0.0001 | 0.0631 | 0.0582 | 0.0879 | 0.0406 | 0.0450 | 0.0548 |
fr | 0.0197 | 1.3204 | 0.2698 | 0.4411 | 1.6306 | 1.7841 | 1.8615 |
gov | 0.3797 | 0.2692 | 1.6174 | 1.5331 | 0.2066 | 0.3140 | 0.4318 |
gr | 0.0014 | 0.1163 | 0.0574 | 0.0765 | 0.4011 | 0.4517 | 0.5808 |
hk | 0.0355 | 0.0431 | 0.0170 | 0.0253 | 0.0823 | 0.0884 | 0.1136 |
hr | 0.0034 | 0.0683 | 0.0136 | 0.0279 | 0.0904 | 0.1280 | 0.1724 |
hu | 0.0025 | 0.3733 | 0.0466 | 0.0971 | 0.4728 | 0.5010 | 0.5531 |
id | 0.0006 | 0.0648 | 0.0556 | 0.0675 | 0.1766 | 0.2239 | 0.3984 |
ie | 0.0017 | 0.1183 | 0.0746 | 0.0920 | 0.1334 | 0.1524 | 0.1994 |
il | 0.0136 | 0.0865 | 0.0216 | 0.0395 | 0.1619 | 0.2088 | 0.1919 |
in | 0.0100 | 0.2150 | 0.1058 | 0.1037 | 0.3898 | 0.4100 | 0.6145 |
info | 1.8065 | 1.3296 | 0.1828 | 0.2468 | 1.4486 | 0.8720 | 0.6747 |
int | 0.0000 | 0.0136 | 0.0739 | 0.0816 | 0.0110 | 0.0234 | 0.0266 |
io | NaN | 0.0049 | 0.0128 | 0.0189 | 0.1329 | 0.1392 | 0.2455 |
ir | 0.0007 | 0.0553 | 0.0141 | 0.0144 | 0.3959 | 0.3380 | 0.3563 |
is | 0.0001 | 0.0345 | 0.0379 | 0.0362 | 0.0378 | 0.0464 | 0.0603 |
it | 0.0439 | 1.3313 | 0.2255 | 0.4406 | 1.5442 | 1.7860 | 2.0514 |
jp | 0.0518 | 1.1750 | 0.1047 | 0.6076 | 2.1181 | 1.9240 | 2.0903 |
kr | 0.0009 | 0.0781 | 0.0451 | 0.0846 | 0.3208 | 0.3337 | 0.3493 |
kz | 0.0003 | 0.0215 | 0.0009 | 0.0096 | 0.1212 | 0.1420 | 0.1252 |
live | NaN | NaN | NaN | 0.0001 | 0.0301 | 0.0924 | 0.0308 |
lt | 0.0003 | 0.0999 | 0.0063 | 0.0218 | 0.1458 | 0.1583 | 0.1667 |
lv | 0.0013 | 0.0610 | 0.0048 | 0.0140 | 0.0901 | 0.1074 | 0.1105 |
me | 0.0000 | 0.0586 | 0.0367 | 0.0557 | 0.3236 | 0.2070 | 0.1747 |
mx | 0.0786 | 0.1461 | 0.0752 | 0.0898 | 0.2027 | 0.2208 | 0.2614 |
my | 0.0302 | 0.0465 | 0.0285 | 0.0414 | 0.0788 | 0.0939 | 0.1217 |
net | 9.2008 | 5.4532 | 3.2465 | 3.4277 | 4.5429 | 3.7758 | 3.4324 |
news | NaN | NaN | NaN | 0.0017 | 0.0313 | 0.0426 | 0.0762 |
ng | 0.0008 | 0.0013 | 0.0008 | 0.0010 | 0.0307 | 0.0420 | 0.0604 |
nl | 0.0073 | 1.4586 | 0.1612 | 0.4427 | 1.3651 | 1.5342 | 1.7284 |
no | 0.0063 | 0.2377 | 0.0612 | 0.0717 | 0.2565 | 0.3199 | 0.3449 |
nu | 0.0001 | 0.0662 | 0.0271 | 0.0208 | 0.0703 | 0.0592 | 0.0580 |
nz | 0.0322 | 0.2276 | 0.0710 | 0.0815 | 0.1964 | 0.2386 | 0.2316 |
online | NaN | NaN | NaN | 0.0013 | 0.0636 | 0.0896 | 0.1403 |
org | 8.1018 | 6.0336 | 9.8274 | 10.5492 | 5.3108 | 5.4622 | 5.8501 |
pe | 0.0155 | 0.0288 | 0.0077 | 0.0153 | 0.0506 | 0.0518 | 0.0866 |
ph | 0.0157 | 0.0316 | 0.0106 | 0.0096 | 0.0392 | 0.0453 | 0.0599 |
pk | 0.0074 | 0.0141 | 0.0109 | 0.0137 | 0.0420 | 0.0434 | 0.0846 |
pl | 0.1857 | 1.8387 | 0.0851 | 0.2666 | 1.5675 | 1.5120 | 1.7593 |
pro | 0.0000 | 0.0082 | 0.0011 | 0.0019 | 0.0718 | 0.0826 | 0.0903 |
pt | 0.0178 | 0.1075 | 0.0414 | 0.0590 | 0.2055 | 0.2472 | 0.3065 |
ro | 0.0191 | 0.5858 | 0.0252 | 0.0918 | 0.4812 | 0.5370 | 0.6493 |
rs | 0.0002 | 0.0389 | 0.0074 | 0.0186 | 0.0819 | 0.1219 | 0.1584 |
ru | 0.0533 | 1.7338 | 0.0892 | 0.6051 | 6.2137 | 6.1853 | 4.5051 |
se | 0.0019 | 0.5331 | 0.1349 | 0.1919 | 0.6682 | 0.6423 | 0.6365 |
sg | 0.0229 | 0.0486 | 0.0190 | 0.0219 | 0.0627 | 0.0701 | 0.0917 |
shop | NaN | NaN | NaN | 0.0000 | 0.0202 | 0.0636 | 0.1468 |
si | 0.0027 | 0.0569 | 0.0071 | 0.0105 | 0.0821 | 0.1146 | 0.1594 |
site | NaN | NaN | NaN | 0.0002 | 0.0426 | 0.0885 | 0.0282 |
sk | 0.0010 | 0.1601 | 0.0107 | 0.0502 | 0.2637 | 0.3205 | 0.3640 |
store | NaN | NaN | NaN | 0.0000 | 0.0078 | 0.0288 | 0.0648 |
su | 0.0001 | 0.0304 | 0.0025 | 0.0052 | 0.1183 | 0.1127 | 0.0783 |
th | 0.0003 | 0.0217 | 0.0206 | 0.0316 | 0.0661 | 0.0991 | 0.1097 |
tk | 0.0000 | 0.0244 | 0.0015 | 0.0021 | 0.0701 | 0.0287 | 0.0040 |
top | NaN | NaN | NaN | 0.0001 | 0.0365 | 0.0570 | 0.0790 |
tr | 0.0623 | 0.1156 | 0.0503 | 0.0769 | 0.2053 | 0.2057 | 0.2836 |
tv | 0.0013 | 0.2584 | 0.1455 | 0.1339 | 0.2500 | 0.2189 | 0.1907 |
tw | 0.0923 | 0.1092 | 0.0505 | 0.0986 | 0.3361 | 0.3920 | 0.3035 |
ua | 0.2190 | 0.4144 | 0.0099 | 0.0538 | 0.9993 | 0.9079 | 0.5302 |
uk | 4.0408 | 3.5037 | 1.5933 | 1.6473 | 2.4210 | 2.3746 | 2.1606 |
us | 0.0046 | 0.4177 | 0.3289 | 0.2700 | 0.3697 | 0.2210 | 0.2109 |
vn | 0.0136 | 0.0797 | 0.0500 | 0.0667 | 0.3862 | 0.4298 | 0.5488 |
ws | 0.0006 | 0.0841 | 0.0109 | 0.0081 | 0.0264 | 0.0198 | 0.0190 |
xyz | NaN | NaN | NaN | 0.0008 | 0.0935 | 0.1379 | 0.1217 |
za | 0.0126 | 0.1614 | 0.0737 | 0.1456 | 0.1982 | 0.2015 | 0.2136 |