Statistics of Common Crawl Monthly Archives

• size of the crawls - number of pages, unique URLs, hosts, domains, top-level domains (public suffixes), cumulative growth of crawled data over time
• top-level domains - distribution and comparison
• crawler-related metrics - fetch status, etc.
• overlaps between monthly crawls
• distribution of
  • media types (MIME)
  • character encodings
  • languages

All metrics presented here can be generated from the Common Crawl URL index by the code of the corresponding project.