Statistics of Common Crawl Monthly Archives

Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives
Latest crawl: CC-MAIN-2024-10

View the Project on GitHub

Top-Level Domains of the Latest Monthly Crawl

Detailed numbers and percentage of top-level domains (groups) in the latest monthly crawl (CC-MAIN-2024-10). Note that Internationalized country code TLDs (IDN ccTLDs) are mapped to their ASCII equivalents before counting TLDs. E.g., the counts of .ru contain also the occurrences of .рф.

pages urls hosts domains %urls %hosts %domains
type
gTLD 1,730,534,374 1,716,434,482 26,492,490 25,182,694 55.71 57.11 57.51
ccTLD 1,308,108,107 1,297,499,274 19,439,479 18,166,637 42.11 41.91 41.49
sTLD 55,645,285 55,117,023 225,379 215,724 1.79 0.49 0.49
grTLD 7,549,900 7,504,696 150,434 144,312 0.24 0.32 0.33
IDN ccTLD 4,343,847 4,326,140 73,937 72,294 0.14 0.16 0.17
IDN gTLD 210,903 210,340 4,810 4,733 0.01 0.01 0.01
ARPA 2 2 2 2 0.00 0.00 0.00

pages urls hosts domains %urls %hosts %domains
tld
com 1,363,820,843 1,352,662,424 21,415,753 20,371,863 43.90 46.17 46.53
org 176,628,223 175,305,520 1,949,544 1,817,146 5.69 4.20 4.15
ru 135,649,457 135,002,702 1,303,469 1,235,591 4.38 2.81 2.82
de 125,699,122 123,447,651 2,878,421 2,657,304 4.01 6.21 6.07
net 103,650,360 102,855,312 1,503,637 1,417,824 3.34 3.24 3.24
uk 65,285,855 64,746,322 1,017,129 947,759 2.10 2.19 2.16
jp 62,997,966 62,640,108 853,262 791,356 2.03 1.84 1.81
it 61,850,597 61,471,929 851,056 801,299 2.00 1.83 1.83
fr 56,185,698 55,781,368 880,808 838,937 1.81 1.90 1.92
pl 53,042,862 52,718,987 690,570 655,307 1.71 1.49 1.50
nl 52,187,517 51,794,001 1,066,748 1,002,660 1.68 2.30 2.29
br 39,208,882 38,991,693 452,750 416,543 1.27 0.98 0.95
edu 35,097,822 34,813,445 129,819 126,628 1.13 0.28 0.29
au 32,470,283 32,252,158 453,444 432,573 1.05 0.98 0.99
es 31,652,617 31,441,024 365,541 345,987 1.02 0.79 0.79
cz 30,361,180 30,126,013 418,012 389,861 0.98 0.90 0.89
ca 29,719,598 29,484,406 442,501 420,546 0.96 0.95 0.96
cn 22,974,413 22,917,928 757,129 684,534 0.74 1.63 1.56
eu 21,400,739 21,252,165 349,086 326,650 0.69 0.75 0.75
info 20,309,656 20,216,824 301,429 285,243 0.66 0.65 0.65
ro 19,537,781 19,458,304 198,587 181,771 0.63 0.43 0.42
se 19,227,530 19,073,628 313,869 295,962 0.62 0.68 0.68
ch 18,943,325 18,777,087 439,401 405,457 0.61 0.95 0.93
in 18,521,850 18,414,707 272,249 256,919 0.60 0.59 0.59
gr 17,479,103 17,405,154 169,236 156,058 0.56 0.36 0.36
hu 16,682,847 16,574,699 205,133 191,413 0.54 0.44 0.44
vn 16,532,462 16,447,016 150,627 143,970 0.53 0.32 0.33
be 16,518,254 16,382,307 337,665 313,795 0.53 0.73 0.72
ua 16,018,353 15,887,928 104,473 99,991 0.52 0.23 0.23
at 14,450,515 14,320,482 332,369 312,672 0.46 0.72 0.71
co 14,029,639 13,648,175 206,671 200,630 0.44 0.45 0.46
gov 13,070,596 12,940,904 30,285 27,759 0.42 0.07 0.06
dk 12,922,729 12,807,203 234,311 221,760 0.42 0.51 0.51
id 11,985,115 11,938,369 137,340 131,972 0.39 0.30 0.30
fi 11,237,335 11,143,601 164,947 154,777 0.36 0.36 0.35
sk 10,987,818 10,907,084 142,336 132,080 0.35 0.31 0.30
ir 10,722,697 10,678,230 109,245 105,308 0.35 0.24 0.24
ar 10,556,508 10,507,214 137,274 120,785 0.34 0.30 0.28
kr 10,508,148 10,468,062 220,953 187,861 0.34 0.48 0.43
no 10,432,840 10,336,580 150,761 142,074 0.34 0.33 0.32
pt 9,246,744 9,185,696 109,797 100,705 0.30 0.24 0.23
tw 9,151,600 9,095,974 107,162 96,678 0.30 0.23 0.22
tr 8,536,008 8,499,523 101,993 92,210 0.28 0.22 0.21
mx 7,879,203 7,832,237 99,478 93,044 0.25 0.21 0.21
io 7,428,003 7,356,123 211,956 209,256 0.24 0.46 0.48
cl 7,017,166 6,978,074 97,148 91,277 0.23 0.21 0.21
nz 6,994,610 6,938,880 105,020 99,888 0.23 0.23 0.23
za 6,441,308 6,401,735 106,175 99,866 0.21 0.23 0.23
us 6,357,615 6,318,824 114,375 109,393 0.21 0.25 0.25
ie 6,025,112 5,976,305 78,975 75,150 0.19 0.17 0.17
il 5,804,909 5,750,699 68,764 65,344 0.19 0.15 0.15
tv 5,760,864 5,713,222 46,059 43,385 0.19 0.10 0.10
by 5,490,086 5,467,570 49,349 47,048 0.18 0.11 0.11
app 5,277,507 5,260,702 79,211 78,482 0.17 0.17 0.18
me 5,279,123 5,234,825 101,499 99,309 0.17 0.22 0.23
hr 5,192,809 5,165,346 54,810 49,191 0.17 0.12 0.11
lt 5,036,216 4,994,062 67,927 62,970 0.16 0.15 0.14
si 4,805,582 4,775,679 59,183 53,480 0.15 0.13 0.12
rs 4,773,917 4,746,893 49,319 44,691 0.15 0.11 0.10
ee 4,634,287 4,602,294 52,573 46,732 0.15 0.11 0.11
bg 4,485,676 4,462,419 33,955 31,685 0.14 0.07 0.07
biz 4,421,247 4,399,399 99,681 95,005 0.14 0.21 0.22
shop 4,435,153 4,399,205 55,635 54,284 0.14 0.12 0.12
online 4,232,118 4,202,972 72,984 71,665 0.14 0.16 0.16
cat 3,935,571 3,896,621 30,424 28,161 0.13 0.07 0.06
kz 3,773,981 3,752,566 37,597 36,027 0.12 0.08 0.08
my 3,670,759 3,646,438 51,527 48,117 0.12 0.11 0.11
xyz 3,666,079 3,645,540 94,390 92,726 0.12 0.20 0.21
cc 3,608,804 3,587,784 55,396 52,099 0.12 0.12 0.12
hk 3,433,047 3,404,169 40,478 35,992 0.11 0.09 0.08
lv 3,341,545 3,310,643 38,863 34,950 0.11 0.08 0.08
th 3,302,880 3,287,609 36,254 32,471 0.11 0.08 0.07
sg 2,769,125 2,746,818 40,106 37,585 0.09 0.09 0.09
pro 2,727,338 2,706,549 44,400 43,283 0.09 0.10 0.10
pe 2,611,695 2,593,839 26,710 25,115 0.08 0.06 0.06
pk 2,551,908 2,536,548 23,365 22,013 0.08 0.05 0.05
top 2,377,962 2,365,864 110,506 104,401 0.08 0.24 0.24
su 2,357,120 2,346,743 23,439 22,295 0.08 0.05 0.05
news 2,291,502 2,283,644 8,557 7,849 0.07 0.02 0.02
club 2,092,617 2,080,210 35,720 34,908 0.07 0.08 0.08
store 1,960,626 1,943,312 27,876 27,320 0.06 0.06 0.06
ng 1,820,300 1,809,576 17,912 17,045 0.06 0.04 0.04
is 1,821,030 1,806,521 19,925 18,649 0.06 0.04 0.04
ph 1,808,211 1,795,125 19,195 18,246 0.06 0.04 0.04
nu 1,751,273 1,738,848 34,124 32,149 0.06 0.07 0.07
ae 1,741,684 1,725,413 21,441 20,248 0.06 0.05 0.05
fm 1,656,415 1,643,548 8,827 8,556 0.05 0.02 0.02