Statistics of Common Crawl Monthly Archives
Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives
Latest crawl: CC-MAIN-2024-46
Groups of Top-Level Domains
The IANA Root Zone Database classifies top-level domains into the following groups:
- generic (“gTLD”), not bound to a specific country. The core is formed by
.com,.info,.netand.orgTLDs, but was later extended to a long list of generic terms (.creditcard,.bike), or brands (.apple,.volkswagen), geographical or cultural entities (.paris,.kiwi) - sponsored TLDs (“sTLD”) are restricted to defined groups of users, registration of domains isn’t open to anybody. That’s obvious for
.gov,.mil,.edubut also applies to.museumand others - generic-restricted (“grTLD”):
.biz,.nameand.pro - one single “infrastructure” TLD:
.arpa - country-code top-level domains:
.uk,.fr,.jp, etc.
The generic and country-code TLDs include internationalized top-level domains written in non-Latin alphabets or containing non-ASCII characters. These are listed here as own groups:
- internationalized country-code TLD (“IDN ccTLD”):
.рф- Russia - internationalized generic TLD (“IDN gTLD”):
.セール- Japanese for ‘sale’,.vermögensberatung- German ‘financial consulting’
Significant changes to the ratio between generic and country-code top-level domains happened in May 2016 (400 million seed donation by Moz.com) and October 2016 where we started to extend the crawl based on host/domain ranks and also increased the crawl size from 1.5–2 billion to about 3 billion pages.
While country-code TLDs have been underrepresented in crawls before 2016, the 30% share of country-code TLDs now comes closer to reality. In the present Alexa top-1-million site list (for details cf. comparison of TLDs) 61% of the TLDs are generic and 38% country-code TLDs.
| type | ARPA | IDN ccTLD | IDN gTLD | ccTLD | gTLD | grTLD | sTLD |
|---|---|---|---|---|---|---|---|
| crawl | |||||||
| CC-MAIN-2008-2009 | NaN | NaN | NaN | 7.49 | 90.30 | 0.55 | 1.66 |
| CC-MAIN-2009-2010 | 0.00 | NaN | NaN | 15.37 | 82.30 | 0.37 | 1.96 |
| CC-MAIN-2012 | 0.00 | 0.00 | NaN | 30.30 | 68.30 | 0.42 | 0.99 |
| CC-MAIN-2013-20 | 0.00 | 0.00 | NaN | 11.89 | 83.69 | 0.06 | 4.36 |
| CC-MAIN-2013-48 | 0.00 | 0.00 | NaN | 7.91 | 87.58 | 0.05 | 4.45 |
| CC-MAIN-2014-10 | 0.00 | 0.00 | NaN | 7.43 | 87.67 | 0.04 | 4.85 |
| CC-MAIN-2014-15 | 0.00 | 0.00 | NaN | 7.89 | 86.91 | 0.05 | 5.15 |
| CC-MAIN-2014-23 | 0.00 | 0.00 | 0.00 | 7.87 | 87.37 | 0.04 | 4.71 |
| CC-MAIN-2014-35 | 0.00 | 0.00 | 0.00 | 7.81 | 87.21 | 0.05 | 4.93 |
| CC-MAIN-2014-41 | 0.00 | 0.00 | 0.00 | 7.77 | 87.22 | 0.05 | 4.96 |
| CC-MAIN-2014-42 | 0.00 | 0.00 | 0.00 | 8.31 | 86.44 | 0.05 | 5.20 |
| CC-MAIN-2014-49 | 0.00 | 0.00 | 0.00 | 8.61 | 86.00 | 0.05 | 5.34 |
| CC-MAIN-2014-52 | NaN | 0.00 | 0.00 | 7.92 | 87.10 | 0.04 | 4.95 |
| CC-MAIN-2015-06 | NaN | 0.00 | 0.00 | 8.02 | 86.86 | 0.04 | 5.09 |
| CC-MAIN-2015-11 | NaN | 0.00 | 0.00 | 7.96 | 86.65 | 0.04 | 5.35 |
| CC-MAIN-2015-14 | NaN | 0.00 | 0.00 | 8.12 | 86.49 | 0.04 | 5.35 |
| CC-MAIN-2015-18 | NaN | 0.00 | 0.00 | 7.87 | 86.99 | 0.03 | 5.11 |
| CC-MAIN-2015-22 | NaN | 0.00 | 0.00 | 7.93 | 86.85 | 0.03 | 5.18 |
| CC-MAIN-2015-27 | NaN | 0.00 | 0.00 | 8.02 | 86.81 | 0.04 | 5.13 |
| CC-MAIN-2015-32 | NaN | 0.00 | 0.00 | 8.00 | 86.82 | 0.04 | 5.14 |
| CC-MAIN-2015-35 | NaN | 0.00 | 0.00 | 8.02 | 86.81 | 0.04 | 5.14 |
| CC-MAIN-2015-40 | NaN | 0.00 | 0.00 | 8.15 | 86.61 | 0.04 | 5.19 |
| CC-MAIN-2015-48 | NaN | 0.00 | 0.00 | 8.09 | 86.70 | 0.04 | 5.17 |
| CC-MAIN-2016-07 | NaN | 0.00 | 0.00 | 8.02 | 86.72 | 0.04 | 5.22 |
| CC-MAIN-2016-18 | 0.00 | 0.00 | 0.00 | 7.93 | 86.72 | 0.05 | 5.30 |
| CC-MAIN-2016-22 | 0.00 | 0.00 | 0.00 | 12.88 | 82.26 | 0.07 | 4.79 |
| CC-MAIN-2016-26 | 0.00 | 0.01 | NaN | 13.06 | 82.32 | 0.08 | 4.54 |
| CC-MAIN-2016-30 | NaN | 0.00 | 0.00 | 12.21 | 83.08 | 0.06 | 4.65 |
| CC-MAIN-2016-36 | 0.00 | 0.00 | 0.00 | 12.89 | 82.44 | 0.06 | 4.60 |
| CC-MAIN-2016-40 | 0.00 | 0.01 | 0.00 | 13.61 | 81.87 | 0.06 | 4.46 |
| CC-MAIN-2016-44 | 0.00 | 0.05 | 0.00 | 30.48 | 66.85 | 0.19 | 2.43 |
| CC-MAIN-2016-50 | 0.00 | 0.04 | 0.00 | 30.52 | 66.95 | 0.22 | 2.27 |
| CC-MAIN-2017-04 | 0.00 | 0.06 | 0.00 | 31.82 | 65.84 | 0.26 | 2.03 |
| CC-MAIN-2017-09 | NaN | 0.04 | 0.00 | 30.84 | 66.91 | 0.23 | 1.98 |
| CC-MAIN-2017-13 | NaN | 0.04 | 0.00 | 31.00 | 67.05 | 0.23 | 1.69 |
| CC-MAIN-2017-17 | 0.00 | 0.03 | 0.00 | 29.61 | 68.22 | 0.24 | 1.89 |
| CC-MAIN-2017-22 | NaN | 0.05 | 0.00 | 30.21 | 67.96 | 0.22 | 1.56 |
| CC-MAIN-2017-26 | NaN | 0.04 | 0.00 | 29.49 | 68.95 | 0.22 | 1.30 |
| CC-MAIN-2017-30 | 0.00 | 0.05 | 0.00 | 29.52 | 68.43 | 0.22 | 1.78 |
| CC-MAIN-2017-34 | NaN | 0.04 | 0.00 | 28.91 | 69.70 | 0.20 | 1.15 |
| CC-MAIN-2017-39 | NaN | 0.07 | 0.00 | 32.93 | 65.26 | 0.23 | 1.50 |
| CC-MAIN-2017-43 | 0.00 | 0.05 | 0.00 | 30.01 | 68.40 | 0.22 | 1.32 |
| CC-MAIN-2017-47 | 0.00 | 0.07 | 0.00 | 33.71 | 64.49 | 0.25 | 1.47 |
| CC-MAIN-2017-51 | 0.00 | 0.06 | 0.00 | 31.44 | 66.94 | 0.25 | 1.31 |
| CC-MAIN-2018-05 | 0.00 | 0.09 | 0.00 | 34.95 | 63.22 | 0.32 | 1.42 |
| CC-MAIN-2018-09 | 0.00 | 0.09 | 0.00 | 33.39 | 64.87 | 0.31 | 1.34 |
| CC-MAIN-2018-13 | 0.00 | 0.08 | 0.00 | 33.98 | 64.38 | 0.27 | 1.28 |
| CC-MAIN-2018-17 | 0.00 | 0.11 | 0.00 | 36.22 | 62.01 | 0.31 | 1.35 |
| CC-MAIN-2018-22 | 0.00 | 0.12 | 0.00 | 36.18 | 62.18 | 0.32 | 1.20 |
| CC-MAIN-2018-26 | 0.00 | 0.10 | 0.00 | 35.37 | 62.90 | 0.30 | 1.33 |
| CC-MAIN-2018-30 | 0.00 | 0.11 | 0.00 | 34.36 | 64.20 | 0.30 | 1.04 |
| CC-MAIN-2018-34 | 0.00 | 0.13 | 0.00 | 38.57 | 59.90 | 0.35 | 1.06 |
| CC-MAIN-2018-39 | 0.00 | 0.12 | 0.00 | 38.67 | 59.76 | 0.33 | 1.11 |
| CC-MAIN-2018-43 | 0.00 | 0.14 | 0.00 | 38.93 | 59.57 | 0.32 | 1.04 |
| CC-MAIN-2018-47 | 0.00 | 0.16 | 0.01 | 39.52 | 59.01 | 0.33 | 0.97 |
| CC-MAIN-2018-51 | 0.00 | 0.14 | 0.01 | 39.63 | 58.87 | 0.31 | 1.03 |
| CC-MAIN-2019-04 | 0.00 | 0.14 | 0.00 | 39.68 | 58.92 | 0.32 | 0.94 |
| CC-MAIN-2019-09 | 0.00 | 0.15 | 0.01 | 40.38 | 58.03 | 0.36 | 1.06 |
| CC-MAIN-2019-13 | 0.00 | 0.13 | 0.01 | 40.41 | 58.02 | 0.32 | 1.11 |
| CC-MAIN-2019-18 | 0.00 | 0.13 | 0.01 | 39.94 | 58.29 | 0.34 | 1.29 |
| CC-MAIN-2019-22 | 0.00 | 0.11 | 0.01 | 39.99 | 58.28 | 0.31 | 1.31 |
| CC-MAIN-2019-26 | 0.00 | 0.12 | 0.01 | 39.65 | 58.45 | 0.30 | 1.49 |
| CC-MAIN-2019-30 | 0.00 | 0.10 | 0.01 | 39.33 | 58.72 | 0.28 | 1.56 |
| CC-MAIN-2019-35 | 0.00 | 0.12 | 0.01 | 39.51 | 58.53 | 0.28 | 1.55 |
| CC-MAIN-2019-39 | 0.00 | 0.10 | 0.01 | 39.44 | 58.35 | 0.26 | 1.84 |
| CC-MAIN-2019-43 | NaN | 0.12 | 0.01 | 39.61 | 58.09 | 0.26 | 1.91 |
| CC-MAIN-2019-47 | NaN | 0.11 | 0.01 | 39.63 | 58.29 | 0.26 | 1.70 |
| CC-MAIN-2019-51 | NaN | 0.12 | 0.00 | 39.36 | 58.51 | 0.25 | 1.75 |
| CC-MAIN-2020-05 | NaN | 0.13 | 0.01 | 39.96 | 58.01 | 0.24 | 1.65 |
| CC-MAIN-2020-10 | NaN | 0.12 | 0.01 | 40.34 | 57.67 | 0.25 | 1.62 |
| CC-MAIN-2020-16 | NaN | 0.14 | 0.01 | 41.18 | 56.95 | 0.24 | 1.48 |
| CC-MAIN-2020-24 | NaN | 0.12 | 0.01 | 40.63 | 57.48 | 0.24 | 1.52 |
| CC-MAIN-2020-29 | NaN | 0.14 | 0.01 | 40.90 | 57.20 | 0.24 | 1.51 |
| CC-MAIN-2020-34 | 0.00 | 0.11 | 0.01 | 40.55 | 57.13 | 0.22 | 1.98 |
| CC-MAIN-2020-40 | 0.00 | 0.19 | 0.01 | 42.21 | 55.81 | 0.27 | 1.51 |
| CC-MAIN-2020-45 | 0.00 | 0.15 | 0.01 | 41.67 | 55.94 | 0.25 | 1.98 |
| CC-MAIN-2020-50 | NaN | 0.16 | 0.01 | 41.66 | 55.96 | 0.26 | 1.96 |
| CC-MAIN-2021-04 | NaN | 0.17 | 0.01 | 41.61 | 56.25 | 0.26 | 1.71 |
| CC-MAIN-2021-10 | NaN | 0.16 | 0.01 | 41.97 | 55.74 | 0.27 | 1.86 |
| CC-MAIN-2021-17 | NaN | 0.17 | 0.01 | 42.18 | 55.64 | 0.27 | 1.74 |
| CC-MAIN-2021-21 | NaN | 0.16 | 0.01 | 41.75 | 55.93 | 0.26 | 1.89 |
| CC-MAIN-2021-25 | NaN | 0.15 | 0.01 | 40.80 | 56.75 | 0.25 | 2.04 |
| CC-MAIN-2021-31 | NaN | 0.16 | 0.01 | 41.20 | 56.70 | 0.27 | 1.66 |
| CC-MAIN-2021-39 | 0.00 | 0.14 | 0.01 | 41.06 | 56.43 | 0.25 | 2.11 |
| CC-MAIN-2021-43 | 0.00 | 0.15 | 0.01 | 41.26 | 56.36 | 0.25 | 1.98 |
| CC-MAIN-2021-49 | NaN | 0.12 | 0.01 | 41.06 | 56.10 | 0.24 | 2.47 |
| CC-MAIN-2022-05 | NaN | 0.13 | 0.01 | 41.40 | 56.07 | 0.25 | 2.15 |
| CC-MAIN-2022-21 | NaN | 0.13 | 0.01 | 40.73 | 56.84 | 0.25 | 2.03 |
| CC-MAIN-2022-27 | NaN | 0.13 | 0.01 | 40.80 | 56.63 | 0.25 | 2.19 |
| CC-MAIN-2022-33 | NaN | 0.12 | 0.01 | 41.09 | 56.02 | 0.24 | 2.53 |
| CC-MAIN-2022-40 | NaN | 0.15 | 0.01 | 37.01 | 60.38 | 0.27 | 2.18 |
| CC-MAIN-2022-49 | NaN | 0.14 | 0.01 | 37.15 | 60.24 | 0.26 | 2.20 |
| CC-MAIN-2023-06 | 0.00 | 0.14 | 0.01 | 37.11 | 60.35 | 0.26 | 2.12 |
| CC-MAIN-2023-14 | 0.00 | 0.15 | 0.01 | 37.47 | 60.09 | 0.25 | 2.03 |
| CC-MAIN-2023-23 | 0.00 | 0.15 | 0.01 | 38.16 | 59.37 | 0.25 | 2.05 |
| CC-MAIN-2023-40 | 0.00 | 0.15 | 0.01 | 38.53 | 59.12 | 0.26 | 1.94 |
| CC-MAIN-2023-50 | 0.00 | 0.17 | 0.01 | 41.94 | 55.71 | 0.25 | 1.92 |
| CC-MAIN-2024-10 | 0.00 | 0.14 | 0.01 | 42.11 | 55.71 | 0.24 | 1.79 |
| CC-MAIN-2024-18 | 0.00 | 0.14 | 0.01 | 41.63 | 56.13 | 0.24 | 1.84 |
| CC-MAIN-2024-22 | 0.00 | 0.14 | 0.01 | 42.43 | 55.34 | 0.23 | 1.84 |
| CC-MAIN-2024-26 | 0.00 | 0.15 | 0.01 | 41.77 | 56.00 | 0.23 | 1.84 |
| CC-MAIN-2024-30 | 0.00 | 0.15 | 0.01 | 42.32 | 55.48 | 0.24 | 1.79 |
| CC-MAIN-2024-33 | 0.00 | 0.14 | 0.01 | 42.37 | 55.34 | 0.25 | 1.90 |
| CC-MAIN-2024-38 | 0.00 | 0.16 | 0.01 | 42.92 | 55.00 | 0.24 | 1.67 |
| CC-MAIN-2024-42 | NaN | 0.17 | 0.01 | 43.22 | 54.57 | 0.24 | 1.80 |
| CC-MAIN-2024-46 | NaN | 0.16 | 0.01 | 42.57 | 55.32 | 0.24 | 1.70 |