Statistics of Common Crawl Monthly Archives

Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives
Latest crawl: CC-MAIN-2024-51

View the Project on GitHub

Top-Level Domains of the Latest Monthly Crawl

Detailed numbers and percentage of top-level domains (groups) in the latest monthly crawl (CC-MAIN-2024-51). Note that Internationalized country code TLDs (IDN ccTLDs) are mapped to their ASCII equivalents before counting TLDs. E.g., the counts of .ru contain also the occurrences of .рф.

pages urls hosts domains %urls %hosts %domains
type
gTLD 1,445,836,917 1,438,259,160 27,768,731 26,608,560 54.86 58.44 58.71
ccTLD 1,131,704,867 1,126,166,290 19,315,607 18,294,650 42.95 40.65 40.36
sTLD 47,352,209 47,038,051 208,952 201,930 1.79 0.44 0.45
grTLD 6,522,291 6,487,866 146,958 142,051 0.25 0.31 0.31
IDN ccTLD 3,683,544 3,671,209 74,580 73,044 0.14 0.16 0.16
IDN gTLD 170,276 169,826 4,069 4,005 0.01 0.01 0.01
ARPA 1 1 1 1 0.00 0.00 0.00

pages urls hosts domains %urls %hosts %domains
tld
com 1,134,968,783 1,128,929,858 22,629,559 21,681,910 43.06 47.62 47.84
org 141,897,876 141,174,556 1,889,967 1,788,015 5.38 3.98 3.94
ru 121,587,996 121,155,750 1,340,862 1,280,165 4.62 2.82 2.82
de 106,134,003 105,533,815 2,812,647 2,634,468 4.03 5.92 5.81
net 89,624,424 89,265,133 1,384,235 1,316,081 3.40 2.91 2.90
jp 59,533,138 59,313,866 846,392 801,855 2.26 1.78 1.77
uk 55,615,208 55,325,904 1,060,555 1,016,333 2.11 2.23 2.24
it 48,205,429 48,010,692 733,820 697,269 1.83 1.54 1.54
pl 47,049,688 46,864,738 685,590 655,747 1.79 1.44 1.45
fr 46,112,908 45,903,724 873,886 840,915 1.75 1.84 1.86
br 41,446,055 41,276,981 560,562 525,602 1.57 1.18 1.16
nl 38,636,218 38,423,907 960,683 913,530 1.47 2.02 2.02
edu 29,262,223 29,132,174 119,454 116,791 1.11 0.25 0.26
au 28,388,987 28,261,490 510,335 495,169 1.08 1.07 1.09
cz 25,867,221 25,680,800 400,115 376,423 0.98 0.84 0.83
cn 24,923,414 24,871,360 952,902 866,703 0.95 2.01 1.91
es 24,532,211 24,415,778 327,983 315,161 0.93 0.69 0.70
ca 23,512,123 23,384,571 422,335 407,388 0.89 0.89 0.90
eu 18,077,606 17,994,829 344,153 326,590 0.69 0.72 0.72
ua 18,010,253 17,686,034 119,560 115,667 0.67 0.25 0.26
in 16,692,041 16,626,793 274,125 261,819 0.63 0.58 0.58
info 16,259,996 16,203,128 258,308 246,124 0.62 0.54 0.54
ro 15,507,025 15,446,739 177,879 169,033 0.59 0.37 0.37
ch 15,491,599 15,406,354 413,792 387,604 0.59 0.87 0.86
se 14,961,635 14,863,250 287,282 274,898 0.57 0.60 0.61
gr 13,957,078 13,910,663 159,062 150,002 0.53 0.33 0.33
be 13,555,831 13,486,668 324,872 305,428 0.51 0.68 0.67
vn 12,728,734 12,677,511 137,513 131,968 0.48 0.29 0.29
hu 12,537,578 12,483,384 184,977 173,876 0.48 0.39 0.38
tr 11,959,499 11,924,528 130,071 119,384 0.45 0.27 0.26
gov 11,579,328 11,502,008 26,291 24,852 0.44 0.06 0.05
co 11,451,582 11,348,585 209,485 204,778 0.43 0.44 0.45
dk 11,280,734 11,184,906 235,986 226,210 0.43 0.50 0.50
at 11,013,704 10,940,460 279,990 265,364 0.42 0.59 0.59
id 10,882,410 10,838,653 148,267 144,151 0.41 0.31 0.32
tw 10,700,652 10,645,059 113,975 103,991 0.41 0.24 0.23
kr 10,540,170 10,512,998 195,234 167,518 0.40 0.41 0.37
ir 10,051,559 10,021,641 107,730 104,073 0.38 0.23 0.23
sk 9,271,621 9,232,604 134,179 125,836 0.35 0.28 0.28
ar 8,847,691 8,822,957 131,108 116,909 0.34 0.28 0.26
fi 8,570,490 8,524,476 148,380 142,221 0.33 0.31 0.31
pt 8,169,926 8,129,865 103,028 96,704 0.31 0.22 0.21
mx 8,104,946 8,071,057 98,979 94,388 0.31 0.21 0.21
no 7,974,176 7,923,425 142,264 136,096 0.30 0.30 0.30
io 6,772,613 6,721,337 208,847 206,527 0.26 0.44 0.46
cl 6,209,602 6,187,247 93,791 89,896 0.24 0.20 0.20
nz 5,964,498 5,937,211 113,969 110,526 0.23 0.24 0.24
za 5,908,615 5,884,406 122,815 118,339 0.22 0.26 0.26
il 5,656,309 5,634,456 84,061 81,301 0.21 0.18 0.18
shop 5,534,569 5,514,326 87,954 86,256 0.21 0.19 0.19
tv 5,052,795 5,018,263 41,418 39,472 0.19 0.09 0.09
by 4,761,665 4,732,605 46,697 44,994 0.18 0.10 0.10
us 4,734,103 4,710,181 101,592 98,212 0.18 0.21 0.22
me 4,710,737 4,648,963 96,527 94,896 0.18 0.20 0.21
lt 4,355,046 4,334,128 66,313 62,293 0.17 0.14 0.14
online 4,292,137 4,267,466 94,469 93,052 0.16 0.20 0.21
hr 4,274,975 4,259,190 52,659 48,682 0.16 0.11 0.11
ie 4,236,638 4,216,292 64,184 61,939 0.16 0.14 0.14
cc 3,987,929 3,952,585 95,037 91,348 0.15 0.20 0.20
ee 3,950,991 3,927,320 50,684 45,683 0.15 0.11 0.10
rs 3,926,008 3,912,295 44,700 41,402 0.15 0.09 0.09
bg 3,903,363 3,889,309 33,476 31,542 0.15 0.07 0.07
app 3,789,980 3,781,233 88,656 87,925 0.14 0.19 0.19
si 3,785,043 3,756,999 55,110 51,048 0.14 0.12 0.11
xyz 3,778,652 3,743,185 131,707 129,905 0.14 0.28 0.29
biz 3,749,880 3,731,272 88,199 84,659 0.14 0.19 0.19
kz 3,383,879 3,372,889 39,022 37,445 0.13 0.08 0.08
th 3,242,643 3,234,078 35,412 32,070 0.12 0.07 0.07
my 3,133,344 3,121,248 48,893 46,691 0.12 0.10 0.10
cat 3,135,417 3,116,947 26,281 24,774 0.12 0.06 0.05
hk 3,056,217 3,037,536 38,371 34,318 0.12 0.08 0.08
lv 2,846,703 2,830,942 34,368 31,535 0.11 0.07 0.07
pro 2,441,544 2,427,887 53,104 52,013 0.09 0.11 0.11
store 2,419,118 2,410,151 39,987 39,350 0.09 0.08 0.09
pe 2,390,467 2,380,714 25,641 24,529 0.09 0.05 0.05
top 2,357,012 2,341,014 112,302 106,812 0.09 0.24 0.24
pk 2,129,763 2,121,792 23,195 22,299 0.08 0.05 0.05
sg 2,129,613 2,120,072 37,500 35,894 0.08 0.08 0.08
news 2,088,088 2,078,029 8,521 7,842 0.08 0.02 0.02
su 2,017,732 2,010,752 22,565 21,608 0.08 0.05 0.05
club 1,692,075 1,680,968 35,075 34,379 0.06 0.07 0.08
ae 1,658,414 1,649,227 24,604 23,659 0.06 0.05 0.05
ai 1,561,810 1,549,702 48,172 47,187 0.06 0.10 0.10
ph 1,548,662 1,542,361 18,854 18,089 0.06 0.04 0.04
is 1,531,850 1,525,159 18,546 17,511 0.06 0.04 0.04
fm 1,503,767 1,499,156 8,795 8,582 0.06 0.02 0.02
ng 1,428,805 1,422,424 17,425 16,824 0.05 0.04 0.04
az 1,358,642 1,354,579 9,834 8,865 0.05 0.02 0.02