Statistics of Common Crawl Monthly Archives

Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives
Latest crawl: CC-MAIN-2024-42

View the Project on GitHub

Top-Level Domains of the Latest Monthly Crawl

Detailed numbers and percentage of top-level domains (groups) in the latest monthly crawl (CC-MAIN-2024-42). Note that Internationalized country code TLDs (IDN ccTLDs) are mapped to their ASCII equivalents before counting TLDs. E.g., the counts of .ru contain also the occurrences of .рф.

pages urls hosts domains %urls %hosts %domains
type
gTLD 1,361,802,018 1,354,443,685 27,795,064 26,612,592 54.57 58.46 58.77
ccTLD 1,077,971,520 1,072,574,953 19,309,814 18,245,143 43.22 40.62 40.29
sTLD 44,912,487 44,587,655 213,339 205,735 1.80 0.45 0.45
grTLD 6,017,403 5,984,570 144,493 139,346 0.24 0.30 0.31
IDN ccTLD 4,178,068 4,167,921 74,851 73,289 0.17 0.16 0.16
IDN gTLD 168,815 168,507 4,161 4,093 0.01 0.01 0.01

pages urls hosts domains %urls %hosts %domains
tld
com 1,060,958,604 1,055,106,611 22,683,163 21,724,670 42.51 47.71 47.98
org 139,506,981 138,790,123 1,844,920 1,737,415 5.59 3.88 3.84
ru 114,560,248 114,188,446 1,328,852 1,268,567 4.60 2.80 2.80
de 98,740,252 98,128,446 2,880,140 2,692,737 3.95 6.06 5.95
net 85,162,485 84,789,193 1,394,430 1,323,267 3.42 2.93 2.92
jp 55,300,351 55,075,500 847,894 799,313 2.22 1.78 1.77
uk 53,499,517 53,223,141 1,054,100 1,005,895 2.14 2.22 2.22
it 47,964,493 47,746,683 757,339 718,185 1.92 1.59 1.59
fr 45,378,809 45,168,507 872,410 837,843 1.82 1.84 1.85
pl 43,053,797 42,866,487 697,700 665,766 1.73 1.47 1.47
br 39,577,206 39,421,304 550,683 516,157 1.59 1.16 1.14
nl 37,214,867 37,005,821 966,799 916,836 1.49 2.03 2.02
edu 28,140,046 27,997,745 122,729 119,967 1.13 0.26 0.26
au 27,148,634 27,024,957 500,265 483,899 1.09 1.05 1.07
cz 26,280,111 26,095,382 416,419 390,960 1.05 0.88 0.86
es 23,711,577 23,598,774 340,865 326,340 0.95 0.72 0.72
ca 22,895,975 22,779,798 421,988 405,689 0.92 0.89 0.90
cn 20,846,580 20,801,518 753,799 677,549 0.84 1.59 1.50
eu 17,435,351 17,345,786 360,587 341,886 0.70 0.76 0.76
ua 16,748,612 16,413,456 116,725 112,629 0.66 0.25 0.25
info 15,744,584 15,689,597 263,718 250,364 0.63 0.55 0.55
in 15,742,023 15,676,152 270,521 257,513 0.63 0.57 0.57
ro 15,215,596 15,166,553 184,256 173,714 0.61 0.39 0.38
ch 15,121,788 15,038,326 422,849 394,994 0.61 0.89 0.87
se 14,150,908 14,065,167 294,566 280,633 0.57 0.62 0.62
gr 13,707,253 13,661,929 161,743 151,790 0.55 0.34 0.34
vn 13,369,416 13,308,555 146,440 140,569 0.54 0.31 0.31
hu 13,203,849 13,146,424 191,765 179,987 0.53 0.40 0.40
be 12,858,042 12,785,572 319,531 299,524 0.52 0.67 0.66
tr 11,363,544 11,336,258 134,629 123,502 0.46 0.28 0.27
dk 11,439,103 11,334,387 246,797 235,963 0.46 0.52 0.52
co 11,099,705 10,981,151 208,497 203,558 0.44 0.44 0.45
id 10,703,155 10,658,581 149,440 145,043 0.43 0.31 0.32
at 10,525,278 10,454,221 280,090 264,519 0.42 0.59 0.58
gov 10,469,080 10,387,876 27,180 25,643 0.42 0.06 0.06
kr 9,582,600 9,552,036 193,289 165,671 0.38 0.41 0.37
ir 9,213,889 9,193,247 106,007 102,534 0.37 0.22 0.23
tw 9,231,050 9,184,629 113,447 103,168 0.37 0.24 0.23
sk 8,890,325 8,848,515 137,900 128,922 0.36 0.29 0.28
fi 8,162,579 8,113,633 151,646 144,598 0.33 0.32 0.32
ar 8,135,763 8,112,498 130,296 116,127 0.33 0.27 0.26
mx 7,842,106 7,809,956 104,990 99,821 0.31 0.22 0.22
no 7,699,423 7,656,980 142,721 136,055 0.31 0.30 0.30
pt 7,612,150 7,577,796 107,697 100,599 0.31 0.23 0.22
io 6,454,777 6,403,601 222,672 220,357 0.26 0.47 0.49
za 6,116,574 6,087,970 129,048 123,962 0.25 0.27 0.27
cl 5,917,407 5,897,289 97,341 92,919 0.24 0.20 0.21
app 5,729,834 5,721,206 105,183 104,443 0.23 0.22 0.23
nz 5,747,372 5,718,163 111,901 108,065 0.23 0.24 0.24
il 5,343,385 5,322,523 82,254 79,230 0.21 0.17 0.17
by 4,907,585 4,880,748 48,032 46,152 0.20 0.10 0.10
tv 4,844,084 4,814,677 40,836 38,766 0.19 0.09 0.09
us 4,615,926 4,592,650 101,204 97,496 0.19 0.21 0.22
me 4,335,328 4,310,803 95,329 93,474 0.17 0.20 0.21
ie 4,316,869 4,293,333 65,773 63,338 0.17 0.14 0.14
lt 4,256,925 4,234,878 66,179 61,973 0.17 0.14 0.14
shop 4,228,867 4,211,550 78,201 76,655 0.17 0.16 0.17
hr 4,101,557 4,088,124 52,525 48,203 0.16 0.11 0.11
xyz 3,990,030 3,966,331 125,197 123,582 0.16 0.26 0.27
ee 3,784,248 3,759,375 50,584 45,532 0.15 0.11 0.10
online 3,777,203 3,758,916 89,371 88,022 0.15 0.19 0.19
bg 3,710,283 3,697,369 34,011 32,017 0.15 0.07 0.07
rs 3,688,324 3,674,549 45,564 41,991 0.15 0.10 0.09
si 3,658,038 3,634,731 56,975 52,401 0.15 0.12 0.12
biz 3,525,111 3,507,128 88,040 84,236 0.14 0.19 0.19
cc 3,437,236 3,408,584 83,672 80,136 0.14 0.18 0.18
kz 3,319,768 3,308,521 39,218 37,608 0.13 0.08 0.08
cat 3,250,065 3,227,823 27,741 26,004 0.13 0.06 0.06
th 2,982,396 2,973,961 35,706 32,286 0.12 0.08 0.07
hk 2,961,745 2,944,061 38,362 34,285 0.12 0.08 0.08
lv 2,890,801 2,877,971 35,401 32,397 0.12 0.07 0.07
my 2,853,681 2,842,421 48,581 46,202 0.11 0.10 0.10
pro 2,166,866 2,154,540 50,589 49,527 0.09 0.11 0.11
pe 2,096,494 2,086,405 25,867 24,673 0.08 0.05 0.05
pk 1,992,950 1,983,808 23,311 22,304 0.08 0.05 0.05
news 1,967,128 1,960,999 8,734 8,018 0.08 0.02 0.02
sg 1,955,905 1,947,250 38,011 36,153 0.08 0.08 0.08
top 1,877,178 1,864,312 119,650 112,408 0.08 0.25 0.25
store 1,838,246 1,830,433 37,306 36,698 0.07 0.08 0.08
su 1,822,618 1,817,201 22,917 21,983 0.07 0.05 0.05
club 1,656,576 1,648,680 36,276 35,544 0.07 0.08 0.08
ng 1,461,047 1,454,135 17,871 17,182 0.06 0.04 0.04
ai 1,458,167 1,447,113 46,217 45,300 0.06 0.10 0.10
ae 1,453,966 1,445,960 23,954 22,957 0.06 0.05 0.05
is 1,421,978 1,414,420 18,952 17,811 0.06 0.04 0.04
ph 1,394,578 1,386,515 18,617 17,836 0.06 0.04 0.04
fm 1,380,953 1,376,323 8,786 8,563 0.06 0.02 0.02
az 1,350,237 1,344,956 9,908 8,881 0.05 0.02 0.02
nu 1,287,977 1,282,088 30,110 28,755 0.05 0.06 0.06