Statistics of Common Crawl Monthly Archives

Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives

View the Project on GitHub

MIME Types

The crawled content is dominated by HTML pages and contains only a small percentage of other document formats. The tables show the percentage of the top 100 media or MIME types of the latest monthly crawls.

While the first table is based the Content-Type HTTP header, the second uses the MIME type detected by Apache Tika based on the actual content.

crawl CC-MAIN-2019-39 CC-MAIN-2019-43 CC-MAIN-2019-47
mimetype
<other> 1.4932 1.2616 0.7756
application/atom+xml 0.1474 0.1066 0.1217
application/download 0.0022 0.0017 0.0019
application/epub+zip 0.0015 0.0010 0.0018
application/force-download 0.0054 0.0044 0.0051
application/javascript 0.0007 0.0004 0.0003
application/json 0.0244 0.0228 0.0227
application/ld+json 0.0004 0.0006 0.0003
application/marc 0.0004 0.0005 0.0004
application/msword 0.0039 0.0026 0.0032
application/octet-stream 0.0529 0.0411 0.0456
application/octetstream 0.0004 0.0002 0.0003
application/pdf 0.9722 0.3764 0.4531
application/pgp-encrypted 0.0001 0.0001 0.0001
application/pgp-signature 0.0015 0.0009 0.0013
application/postscript 0.0009 0.0007 0.0006
application/rdf+xml 0.0030 0.0032 0.0026
application/rss+xml 0.0855 0.0743 0.0737
application/rtf 0.0010 0.0013 0.0011
application/save-to-disk 0.0000 0.0000 0.0000
application/text 0.0033 0.0032 0.0029
application/unknown 0.0004 0.0002 0.0003
application/vnd.android.package-archive 0.0010 0.0011 0.0011
application/vnd.google-earth.kml+xml 0.0017 0.0015 0.0016
application/vnd.google-earth.kmz 0.0003 0.0003 0.0003
application/vnd.ms-excel 0.0017 0.0012 0.0013
application/vnd.ms-powerpoint 0.0008 0.0011 0.0016
application/vnd.ms-word 0.0007 0.0007 0.0007
application/vnd.ms-word.document.macroenabled.12 0.0000 0.0000 0.0000
application/vnd.oasis.opendocument.text 0.0008 0.0006 0.0007
application/vnd.openxmlformats-officedocument.presentationml.presentation 0.0003 0.0002 0.0002
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet 0.0013 0.0010 0.0012
application/vnd.openxmlformats-officedocument.wordprocessingml.document 0.0025 0.0022 0.0023
application/vnd.wap.xhtml+xml 0.0001 0.0001 0.0001
application/x-bibtex 0.0020 0.0022 0.0018
application/x-bittorrent 0.0004 0.0005 0.0003
application/x-debian-package 0.0000 0.0001 0.0002
application/x-download 0.0014 0.0011 0.0011
application/x-dvi 0.0000 0.0000 0.0000
application/x-endnote-refer 0.0038 0.0042 0.0036
application/x-gzip 0.0012 0.0011 0.0009
application/x-httpd-php 0.0006 0.0006 0.0007
application/x-java-jnlp-file 0.0002 0.0001 0.0003
application/x-javascript 0.0003 0.0005 0.0003
application/x-json 0.0000 0.0000 0.0000
application/x-mobipocket-ebook 0.0001 0.0001 0.0002
application/x-msdownload 0.0005 0.0003 0.0003
application/x-netcdf 0.0001 0.0003 0.0002
application/x-research-info-systems 0.0059 0.0074 0.0063
application/x-shockwave-flash 0.0008 0.0006 0.0005
application/x-tar 0.0004 0.0001 0.0002
application/x-tex 0.0001 0.0000 0.0001
application/x-troff-man 0.0005 0.0006 0.0003
application/x-xz 0.0000 0.0000 0.0000
application/x-zip-compressed 0.0007 0.0005 0.0007
application/xhtml+xml 0.0443 0.0522 0.0401
application/xml 0.0321 0.0298 0.0265
application/zip 0.0047 0.0031 0.0040
audio/mp3 0.0004 0.0002 0.0002
audio/mp4 0.0006 0.0006 0.0008
audio/mpeg 0.0047 0.0032 0.0044
audio/x-mpegurl 0.0010 0.0010 0.0009
audio/x-pn-realaudio 0.0001 0.0000 0.0001
audio/x-scpls 0.0001 0.0001 0.0001
audio/x-wav 0.0006 0.0007 0.0008
binary/octet-stream 0.0010 0.0005 0.0016
image/gif 0.0030 0.0026 0.0027
image/jp2 0.0003 0.0002 0.0002
image/jpeg 0.1565 0.1163 0.1410
image/jpg 0.0029 0.0023 0.0027
image/pjpeg 0.0006 0.0004 0.0005
image/png 0.0244 0.0182 0.0219
image/svg+xml 0.0005 0.0004 0.0006
image/tiff 0.0005 0.0004 0.0004
image/vnd.djvu 0.0003 0.0003 0.0003
image/webp 0.0007 0.0010 0.0007
message/rfc822 0.0005 0.0004 0.0004
text/calendar 0.0424 0.0340 0.0356
text/csv 0.0024 0.0031 0.0028
text/enriched 0.0012 0.0013 0.0013
text/html 96.6921 97.6457 98.0216
text/javascript 0.0005 0.0004 0.0005
text/pdf 0.0001 0.0000 0.0000
text/plain 0.0579 0.0602 0.0536
text/tab-separated-values 0.0006 0.0006 0.0006
text/turtle 0.0006 0.0008 0.0007
text/vnd.wap.wml 0.0002 0.0002 0.0002
text/x-bibtex 0.0003 0.0003 0.0002
text/x-c 0.0001 0.0001 0.0001
text/x-csrc 0.0008 0.0013 0.0008
text/x-diff 0.0003 0.0002 0.0003
text/x-patch 0.0003 0.0004 0.0004
text/x-perl 0.0001 0.0001 0.0001
text/x-vcalendar 0.0009 0.0006 0.0006
text/x-vcard 0.0023 0.0020 0.0019
text/xml 0.0834 0.0789 0.0819
unknown/unknown 0.0007 0.0006 0.0006
video/mp4 0.0015 0.0010 0.0014
video/webm 0.0001 0.0001 0.0001
video/x-ms-asf 0.0003 0.0003 0.0001

crawl CC-MAIN-2019-39 CC-MAIN-2019-43 CC-MAIN-2019-47
mimetype_detected
<other> 0.0140 0.0127 0.0135
application/atom+xml 0.1527 0.1116 0.1259
application/epub+zip 0.0018 0.0013 0.0022
application/force-download 0.0002 0.0003 0.0001
application/gzip 0.0023 0.0022 0.0027
application/javascript 0.0009 0.0011 0.0011
application/json 0.0246 0.0238 0.0245
application/ld+json nan 0.0000 nan
application/marc 0.0003 0.0003 0.0002
application/mbox 0.0022 0.0021 0.0024
application/msword 0.0032 0.0021 0.0027
application/octet-stream 0.0174 0.0159 0.0127
application/pdf 0.9968 0.3902 0.4712
application/pgp-signature 0.0018 0.0011 0.0015
application/pls+xml 0.0001 0.0001 0.0001
application/postscript 0.0008 0.0005 0.0005
application/rdf+xml 0.0127 0.0108 0.0093
application/rss+xml 0.1507 0.1371 0.1322
application/rtf 0.0027 0.0029 0.0027
application/text 0.0005 0.0007 0.0005
application/vnd.android.package-archive 0.0014 0.0012 0.0015
application/vnd.google-earth.kml+xml 0.0023 0.0019 0.0021
application/vnd.google-earth.kmz 0.0003 0.0003 0.0004
application/vnd.ms-excel 0.0014 0.0010 0.0012
application/vnd.ms-powerpoint 0.0008 0.0011 0.0016
application/vnd.oasis.opendocument.text 0.0009 0.0006 0.0007
application/vnd.openxmlformats-officedocument.presentationml.presentation 0.0003 0.0002 0.0002
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet 0.0013 0.0010 0.0012
application/vnd.openxmlformats-officedocument.wordprocessingml.document 0.0025 0.0023 0.0024
application/x-bibtex-text-file 0.0044 0.0050 0.0049
application/x-bittorrent 0.0005 0.0005 0.0003
application/x-bzip2 0.0007 0.0004 0.0007
application/x-debian-package 0.0001 0.0001 0.0002
application/x-dosexec 0.0006 0.0004 0.0005
application/x-endnote-refer 0.0019 0.0021 0.0019
application/x-mobipocket-ebook 0.0003 0.0002 0.0004
application/x-ms-asx 0.0003 0.0003 0.0001
application/x-msdownload 0.0002 0.0001 0.0001
application/x-rar-compressed 0.0007 0.0004 0.0005
application/x-research-info-systems 0.0001 0.0002 0.0002
application/x-sh 0.0011 0.0008 0.0013
application/x-shockwave-flash 0.0008 0.0006 0.0005
application/x-stata-do 0.0003 0.0003 0.0002
application/x-tika-msoffice 0.0034 0.0027 0.0029
application/x-tika-ooxml 0.0027 0.0019 0.0019
application/x-wais-source 0.0002 0.0001 0.0002
application/x-xz 0.0005 0.0005 0.0002
application/xhtml+xml 21.9389 22.3992 21.4491
application/xml 0.0594 0.0615 0.0661
application/zip 0.0088 0.0064 0.0086
application/zlib 0.0012 0.0008 0.0004
audio/midi 0.0001 0.0001 0.0001
audio/mp4 0.0014 0.0011 0.0014
audio/mpeg 0.0066 0.0046 0.0057
audio/vnd.wave 0.0008 0.0009 0.0009
audio/vorbis 0.0003 0.0001 0.0001
audio/x-mpegurl 0.0001 0.0002 0.0001
image/bmp 0.0001 0.0001 0.0001
image/gif 0.0031 0.0027 0.0028
image/jp2 0.0003 0.0002 0.0002
image/jpeg 0.1673 0.1245 0.1518
image/png 0.0245 0.0186 0.0225
image/svg+xml 0.0006 0.0004 0.0007
image/tiff 0.0013 0.0010 0.0021
image/vnd.djvu 0.0005 0.0004 0.0005
image/vnd.dxf; format=ascii 0.0004 0.0003 0.0003
image/vnd.microsoft.icon 0.0004 0.0005 0.0005
image/webp 0.0006 0.0009 0.0009
message/rfc822 0.0014 0.0012 0.0012
text/asp 0.0000 nan nan
text/calendar 0.0442 0.0353 0.0370
text/css 0.0002 0.0003 0.0004
text/csv 0.0023 0.0028 0.0026
text/html 76.2041 76.4787 77.3011
text/plain 0.0852 0.0838 0.0799
text/prs.lines.tag 0.0014 0.0015 0.0016
text/tab-separated-values 0.0006 0.0006 0.0006
text/troff 0.0004 0.0005 0.0004
text/turtle 0.0006 0.0008 0.0007
text/x-c++src 0.0004 0.0004 0.0004
text/x-chdr 0.0011 0.0015 0.0011
text/x-csrc 0.0018 0.0025 0.0018
text/x-diff 0.0020 0.0027 0.0021
text/x-jsp 0.0001 0.0001 0.0002
text/x-log 0.0038 0.0034 0.0029
text/x-matlab 0.0022 0.0026 0.0021
text/x-patch 0.0001 0.0002 0.0002
text/x-perl 0.0032 0.0038 0.0031
text/x-php 0.0034 0.0032 0.0034
text/x-python 0.0003 0.0003 0.0003
text/x-vcalendar 0.0010 0.0008 0.0007
text/x-vcard 0.0040 0.0034 0.0037
video/mp4 0.0017 0.0012 0.0016
video/quicktime 0.0004 0.0003 0.0004
video/webm 0.0001 0.0001 0.0001
video/x-m4v 0.0005 0.0004 0.0006
video/x-matroska 0.0004 0.0002 0.0002
video/x-ms-wmv 0.0001 0.0001 0.0001