Statistics of Common Crawl Monthly Archives

Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives
Latest crawl: CC-MAIN-2024-10

View the Project on GitHub

Groups of Top-Level Domains

The IANA Root Zone Database classifies top-level domains into the following groups:

The generic and country-code TLDs include internationalized top-level domains written in non-Latin alphabets or containing non-ASCII characters. These are listed here as own groups:

Percentage of top-level domains groups

Significant changes to the ratio between generic and country-code top-level domains happened in May 2016 (400 million seed donation by Moz.com) and October 2016 where we started to extend the crawl based on host/domain ranks and also increased the crawl size from 1.5–2 billion to about 3 billion pages.

While country-code TLDs have been underrepresented in crawls before 2016, the 30% share of country-code TLDs now comes closer to reality. In the present Alexa top-1-million site list (for details cf. comparison of TLDs) 61% of the TLDs are generic and 38% country-code TLDs.

type ARPA IDN ccTLD IDN gTLD ccTLD gTLD grTLD sTLD
crawl
CC-MAIN-2008-2009 NaN NaN NaN 7.49 90.30 0.55 1.66
CC-MAIN-2009-2010 0.00 NaN NaN 15.37 82.30 0.37 1.96
CC-MAIN-2012 0.00 0.00 NaN 30.30 68.30 0.42 0.99
CC-MAIN-2013-20 0.00 0.00 NaN 11.89 83.69 0.06 4.36
CC-MAIN-2013-48 0.00 0.00 NaN 7.91 87.58 0.05 4.45
CC-MAIN-2014-10 0.00 0.00 NaN 7.43 87.67 0.04 4.85
CC-MAIN-2014-15 0.00 0.00 NaN 7.89 86.91 0.05 5.15
CC-MAIN-2014-23 0.00 0.00 0.00 7.87 87.37 0.04 4.71
CC-MAIN-2014-35 0.00 0.00 0.00 7.81 87.21 0.05 4.93
CC-MAIN-2014-41 0.00 0.00 0.00 7.77 87.22 0.05 4.96
CC-MAIN-2014-42 0.00 0.00 0.00 8.31 86.44 0.05 5.20
CC-MAIN-2014-49 0.00 0.00 0.00 8.61 86.00 0.05 5.34
CC-MAIN-2014-52 NaN 0.00 0.00 7.92 87.10 0.04 4.95
CC-MAIN-2015-06 NaN 0.00 0.00 8.02 86.86 0.04 5.09
CC-MAIN-2015-11 NaN 0.00 0.00 7.96 86.65 0.04 5.35
CC-MAIN-2015-14 NaN 0.00 0.00 8.12 86.49 0.04 5.35
CC-MAIN-2015-18 NaN 0.00 0.00 7.87 86.99 0.03 5.11
CC-MAIN-2015-22 NaN 0.00 0.00 7.93 86.85 0.03 5.18
CC-MAIN-2015-27 NaN 0.00 0.00 8.02 86.81 0.04 5.13
CC-MAIN-2015-32 NaN 0.00 0.00 8.00 86.82 0.04 5.14
CC-MAIN-2015-35 NaN 0.00 0.00 8.02 86.81 0.04 5.14
CC-MAIN-2015-40 NaN 0.00 0.00 8.15 86.61 0.04 5.19
CC-MAIN-2015-48 NaN 0.00 0.00 8.09 86.70 0.04 5.17
CC-MAIN-2016-07 NaN 0.00 0.00 8.02 86.72 0.04 5.22
CC-MAIN-2016-18 0.00 0.00 0.00 7.93 86.72 0.05 5.30
CC-MAIN-2016-22 0.00 0.00 0.00 12.88 82.26 0.07 4.79
CC-MAIN-2016-26 0.00 0.01 NaN 13.06 82.32 0.08 4.54
CC-MAIN-2016-30 NaN 0.00 0.00 12.21 83.08 0.06 4.65
CC-MAIN-2016-36 0.00 0.00 0.00 12.89 82.44 0.06 4.60
CC-MAIN-2016-40 0.00 0.01 0.00 13.61 81.87 0.06 4.46
CC-MAIN-2016-44 0.00 0.05 0.00 30.48 66.85 0.19 2.43
CC-MAIN-2016-50 0.00 0.04 0.00 30.52 66.95 0.22 2.27
CC-MAIN-2017-04 0.00 0.06 0.00 31.82 65.84 0.26 2.03
CC-MAIN-2017-09 NaN 0.04 0.00 30.84 66.91 0.23 1.98
CC-MAIN-2017-13 NaN 0.04 0.00 31.00 67.05 0.23 1.69
CC-MAIN-2017-17 0.00 0.03 0.00 29.61 68.22 0.24 1.89
CC-MAIN-2017-22 NaN 0.05 0.00 30.21 67.96 0.22 1.56
CC-MAIN-2017-26 NaN 0.04 0.00 29.49 68.95 0.22 1.30
CC-MAIN-2017-30 0.00 0.05 0.00 29.52 68.43 0.22 1.78
CC-MAIN-2017-34 NaN 0.04 0.00 28.91 69.70 0.20 1.15
CC-MAIN-2017-39 NaN 0.07 0.00 32.93 65.26 0.23 1.50
CC-MAIN-2017-43 0.00 0.05 0.00 30.01 68.40 0.22 1.32
CC-MAIN-2017-47 0.00 0.07 0.00 33.71 64.49 0.25 1.47
CC-MAIN-2017-51 0.00 0.06 0.00 31.44 66.94 0.25 1.31
CC-MAIN-2018-05 0.00 0.09 0.00 34.95 63.22 0.32 1.42
CC-MAIN-2018-09 0.00 0.09 0.00 33.39 64.87 0.31 1.34
CC-MAIN-2018-13 0.00 0.08 0.00 33.98 64.38 0.27 1.28
CC-MAIN-2018-17 0.00 0.11 0.00 36.22 62.01 0.31 1.35
CC-MAIN-2018-22 0.00 0.12 0.00 36.18 62.18 0.32 1.20
CC-MAIN-2018-26 0.00 0.10 0.00 35.37 62.90 0.30 1.33
CC-MAIN-2018-30 0.00 0.11 0.00 34.36 64.20 0.30 1.04
CC-MAIN-2018-34 0.00 0.13 0.00 38.57 59.90 0.35 1.06
CC-MAIN-2018-39 0.00 0.12 0.00 38.67 59.76 0.33 1.11
CC-MAIN-2018-43 0.00 0.14 0.00 38.93 59.57 0.32 1.04
CC-MAIN-2018-47 0.00 0.16 0.01 39.52 59.01 0.33 0.97
CC-MAIN-2018-51 0.00 0.14 0.01 39.63 58.87 0.31 1.03
CC-MAIN-2019-04 0.00 0.14 0.00 39.68 58.92 0.32 0.94
CC-MAIN-2019-09 0.00 0.15 0.01 40.38 58.03 0.36 1.06
CC-MAIN-2019-13 0.00 0.13 0.01 40.41 58.02 0.32 1.11
CC-MAIN-2019-18 0.00 0.13 0.01 39.94 58.29 0.34 1.29
CC-MAIN-2019-22 0.00 0.11 0.01 39.99 58.28 0.31 1.31
CC-MAIN-2019-26 0.00 0.12 0.01 39.65 58.45 0.30 1.49
CC-MAIN-2019-30 0.00 0.10 0.01 39.33 58.72 0.28 1.56
CC-MAIN-2019-35 0.00 0.12 0.01 39.51 58.53 0.28 1.55
CC-MAIN-2019-39 0.00 0.10 0.01 39.44 58.35 0.26 1.84
CC-MAIN-2019-43 NaN 0.12 0.01 39.61 58.09 0.26 1.91
CC-MAIN-2019-47 NaN 0.11 0.01 39.63 58.29 0.26 1.70
CC-MAIN-2019-51 NaN 0.12 0.00 39.36 58.51 0.25 1.75
CC-MAIN-2020-05 NaN 0.13 0.01 39.96 58.01 0.24 1.65
CC-MAIN-2020-10 NaN 0.12 0.01 40.34 57.67 0.25 1.62
CC-MAIN-2020-16 NaN 0.14 0.01 41.18 56.95 0.24 1.48
CC-MAIN-2020-24 NaN 0.12 0.01 40.63 57.48 0.24 1.52
CC-MAIN-2020-29 NaN 0.14 0.01 40.90 57.20 0.24 1.51
CC-MAIN-2020-34 0.00 0.11 0.01 40.55 57.13 0.22 1.98
CC-MAIN-2020-40 0.00 0.19 0.01 42.21 55.81 0.27 1.51
CC-MAIN-2020-45 0.00 0.15 0.01 41.67 55.94 0.25 1.98
CC-MAIN-2020-50 NaN 0.16 0.01 41.66 55.96 0.26 1.96
CC-MAIN-2021-04 NaN 0.17 0.01 41.61 56.25 0.26 1.71
CC-MAIN-2021-10 NaN 0.16 0.01 41.97 55.74 0.27 1.86
CC-MAIN-2021-17 NaN 0.17 0.01 42.18 55.64 0.27 1.74
CC-MAIN-2021-21 NaN 0.16 0.01 41.75 55.93 0.26 1.89
CC-MAIN-2021-25 NaN 0.15 0.01 40.80 56.75 0.25 2.04
CC-MAIN-2021-31 NaN 0.16 0.01 41.20 56.70 0.27 1.66
CC-MAIN-2021-39 0.00 0.14 0.01 41.06 56.43 0.25 2.11
CC-MAIN-2021-43 0.00 0.15 0.01 41.26 56.36 0.25 1.98
CC-MAIN-2021-49 NaN 0.12 0.01 41.06 56.10 0.24 2.47
CC-MAIN-2022-05 NaN 0.13 0.01 41.40 56.07 0.25 2.15
CC-MAIN-2022-21 NaN 0.13 0.01 40.73 56.84 0.25 2.03
CC-MAIN-2022-27 NaN 0.13 0.01 40.80 56.63 0.25 2.19
CC-MAIN-2022-33 NaN 0.12 0.01 41.09 56.02 0.24 2.53
CC-MAIN-2022-40 NaN 0.15 0.01 37.01 60.38 0.27 2.18
CC-MAIN-2022-49 NaN 0.14 0.01 37.15 60.24 0.26 2.20
CC-MAIN-2023-06 0.00 0.14 0.01 37.11 60.35 0.26 2.12
CC-MAIN-2023-14 0.00 0.15 0.01 37.47 60.09 0.25 2.03
CC-MAIN-2023-23 0.00 0.15 0.01 38.16 59.37 0.25 2.05
CC-MAIN-2023-40 0.00 0.15 0.01 38.53 59.12 0.26 1.94
CC-MAIN-2023-50 0.00 0.17 0.01 41.94 55.71 0.25 1.92
CC-MAIN-2024-10 0.00 0.14 0.01 42.11 55.71 0.24 1.79