Statistics of Common Crawl Monthly Archives

Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives

View the Project on GitHub

MIME Types

The crawled content is dominated by HTML pages and contains only a small percentage of other document formats. The tables show the percentage of the top 100 media or MIME types of the latest monthly crawls.

While the first table is based the Content-Type HTTP header, the second uses the MIME type detected by Apache Tika based on the actual content. The underlying data including page counts is provided in mimetypes.csv resp. mimetypes_detected.csv.

crawl CC-MAIN-2020-16 CC-MAIN-2020-24 CC-MAIN-2020-29
mimetype
<other> 0.8410 1.0384 0.2046
application/atom+xml 0.0917 0.1571 0.1241
application/download 0.0023 0.0031 0.0027
application/epub+zip 0.0013 0.0013 0.0011
application/force-download 0.0042 0.0059 0.0055
application/javascript 0.0004 0.0013 0.0009
application/json 0.0166 0.0206 0.0187
application/ld+json 0.0003 0.0005 0.0004
application/marc 0.0004 0.0005 0.0004
application/msword 0.0027 0.0028 0.0030
application/octet-stream 0.0364 0.0400 0.0389
application/octetstream 0.0002 0.0002 0.0001
application/pdf 0.7246 0.1160 0.0900
application/pgp-encrypted 0.0000 0.0000 0.0000
application/pgp-signature 0.0013 0.0009 0.0011
application/postscript 0.0002 0.0002 0.0002
application/rdf+xml 0.0022 0.0027 0.0025
application/rss+xml 0.0717 0.0824 0.0773
application/rtf 0.0011 0.0013 0.0012
application/save-to-disk 0.0000 0.0000 0.0000
application/text 0.0022 0.0028 0.0024
application/unknown 0.0003 0.0004 0.0003
application/vnd.android.package-archive 0.0010 0.0010 0.0014
application/vnd.google-earth.kml+xml 0.0013 0.0017 0.0018
application/vnd.google-earth.kmz 0.0003 0.0004 0.0003
application/vnd.ms-excel 0.0014 0.0015 0.0013
application/vnd.ms-powerpoint 0.0007 0.0009 0.0007
application/vnd.ms-word 0.0005 0.0008 0.0007
application/vnd.ms-word.document.macroenabled.12 0.0000 0.0000 0.0000
application/vnd.oasis.opendocument.text 0.0008 0.0009 0.0009
application/vnd.openxmlformats-officedocument.presentationml.presentation 0.0002 0.0002 0.0002
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet 0.0010 0.0014 0.0011
application/vnd.openxmlformats-officedocument.wordprocessingml.document 0.0020 0.0026 0.0025
application/vnd.wap.xhtml+xml 0.0001 0.0001 0.0001
application/x-bibtex 0.0015 0.0020 0.0018
application/x-bittorrent 0.0003 0.0004 0.0004
application/x-debian-package 0.0003 0.0001 0.0000
application/x-download 0.0012 0.0012 0.0012
application/x-endnote-refer 0.0027 0.0034 0.0031
application/x-gzip 0.0009 0.0008 0.0006
application/x-httpd-php 0.0005 0.0007 0.0005
application/x-java-jnlp-file 0.0001 0.0003 0.0003
application/x-javascript 0.0002 0.0005 0.0004
application/x-json 0.0000 0.0000 0.0000
application/x-mobipocket-ebook 0.0001 0.0001 0.0001
application/x-msdownload 0.0003 0.0003 0.0003
application/x-netcdf 0.0001 0.0001 0.0002
application/x-research-info-systems 0.0053 0.0060 0.0059
application/x-shockwave-flash 0.0003 0.0003 0.0002
application/x-tar 0.0002 0.0002 0.0002
application/x-tex 0.0001 0.0001 0.0001
application/x-troff-man 0.0003 0.0001 0.0004
application/x-xz 0.0001 0.0000 0.0000
application/x-zip-compressed 0.0006 0.0007 0.0005
application/xhtml+xml 0.0359 0.0400 0.0391
application/xml 0.0292 0.0302 0.0283
application/zip 0.0037 0.0034 0.0030
audio/mp3 0.0002 0.0002 0.0002
audio/mp4 0.0006 0.0007 0.0007
audio/mpeg 0.0043 0.0041 0.0035
audio/x-mpegurl 0.0008 0.0010 0.0007
audio/x-pn-realaudio 0.0001 0.0001 0.0001
audio/x-scpls 0.0001 0.0001 0.0001
audio/x-wav 0.0004 0.0002 0.0001
binary/octet-stream 0.0007 0.0006 0.0005
image/gif 0.0024 0.0022 0.0021
image/jp2 0.0001 0.0000 0.0000
image/jpeg 0.1211 0.1148 0.1202
image/jpg 0.0026 0.0021 0.0021
image/pjpeg 0.0004 0.0005 0.0003
image/png 0.0198 0.0173 0.0188
image/svg+xml 0.0004 0.0004 0.0003
image/tiff 0.0002 0.0003 0.0003
image/vnd.djvu 0.0003 0.0002 0.0002
image/webp 0.0007 0.0012 0.0010
message/rfc822 0.0004 0.0006 0.0007
text/calendar 0.0311 0.0472 0.0430
text/csv 0.0027 0.0034 0.0030
text/directory 0.0004 0.0004 0.0005
text/enriched 0.0007 0.0010 0.0010
text/html 97.7633 98.0659 98.9806
text/javascript 0.0005 0.0005 0.0003
text/pdf 0.0000 0.0000 0.0000
text/plain 0.0514 0.0488 0.0486
text/tab-separated-values 0.0004 0.0007 0.0005
text/turtle 0.0009 0.0010 0.0009
text/vcard 0.0008 0.0011 0.0009
text/vnd.wap.wml 0.0003 0.0002 0.0002
text/x-bibtex 0.0003 0.0002 0.0003
text/x-c 0.0001 0.0001 0.0001
text/x-csrc 0.0007 0.0006 0.0005
text/x-diff 0.0002 0.0002 0.0002
text/x-patch 0.0004 0.0004 0.0004
text/x-perl 0.0002 0.0001 0.0002
text/x-vcalendar 0.0006 0.0009 0.0007
text/x-vcard 0.0017 0.0023 0.0021
text/xml 0.0915 0.0951 0.0894
unknown/unknown 0.0006 0.0005 0.0004
video/mp4 0.0012 0.0012 0.0011
video/webm 0.0001 0.0001 0.0001
video/x-ms-asf 0.0002 0.0002 0.0002

crawl CC-MAIN-2020-16 CC-MAIN-2020-24 CC-MAIN-2020-29
mimetype_detected
<other> 0.0126 0.0135 0.0133
application/atom+xml 0.0966 0.1622 0.1291
application/epub+zip 0.0016 0.0016 0.0014
application/force-download 0.0001 0.0002 0.0001
application/gzip 0.0025 0.0030 0.0032
application/javascript 0.0011 0.0020 0.0014
application/json 0.0164 0.0206 0.0182
application/marc 0.0002 0.0003 0.0003
application/mbox 0.0019 0.0023 0.0022
application/msword 0.0022 0.0021 0.0024
application/octet-stream 0.0092 0.0103 0.0108
application/pdf 0.7425 0.1269 0.0993
application/pgp-signature 0.0016 0.0012 0.0013
application/pls+xml 0.0002 0.0001 0.0001
application/postscript 0.0001 0.0002 0.0002
application/rdf+xml 0.0100 0.0099 0.0094
application/rss+xml 0.1300 0.1541 0.1426
application/rtf 0.0021 0.0024 0.0022
application/text 0.0005 0.0006 0.0006
application/vnd.android.package-archive 0.0014 0.0013 0.0014
application/vnd.google-earth.kml+xml 0.0017 0.0023 0.0023
application/vnd.google-earth.kmz 0.0003 0.0004 0.0003
application/vnd.ms-excel 0.0011 0.0012 0.0010
application/vnd.ms-powerpoint 0.0008 0.0009 0.0007
application/vnd.oasis.opendocument.text 0.0006 0.0008 0.0007
application/vnd.openxmlformats-officedocument.presentationml.presentation 0.0002 0.0002 0.0002
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet 0.0010 0.0014 0.0011
application/vnd.openxmlformats-officedocument.wordprocessingml.document 0.0021 0.0026 0.0025
application/x-bibtex-text-file 0.0050 0.0057 0.0054
application/x-bittorrent 0.0003 0.0005 0.0004
application/x-bzip2 0.0002 0.0002 0.0004
application/x-debian-package 0.0004 0.0003 0.0002
application/x-dosexec 0.0004 0.0003 0.0003
application/x-endnote-refer 0.0015 0.0021 0.0018
application/x-mobipocket-ebook 0.0003 0.0003 0.0003
application/x-ms-asx 0.0002 0.0002 0.0001
application/x-msdownload 0.0001 0.0001 0.0002
application/x-rar-compressed 0.0005 0.0000 0.0000
application/x-research-info-systems 0.0002 0.0002 0.0002
application/x-sh 0.0009 0.0008 0.0006
application/x-shockwave-flash 0.0003 0.0003 0.0002
application/x-stata-do 0.0002 0.0005 0.0003
application/x-tika-msoffice 0.0025 0.0036 0.0033
application/x-tika-ooxml 0.0019 0.0028 0.0028
application/x-wais-source 0.0002 0.0001 0.0004
application/x-xz 0.0002 0.0001 0.0001
application/xhtml+xml 20.1664 20.8330 19.9205
application/xml 0.0842 0.0746 0.0700
application/zip 0.0076 0.0071 0.0062
application/zlib 0.0002 0.0002 0.0002
audio/midi 0.0001 0.0001 0.0001
audio/mp4 0.0015 0.0015 0.0014
audio/mpeg 0.0056 0.0051 0.0044
audio/vnd.wave 0.0008 0.0006 0.0004
audio/vorbis 0.0001 0.0000 0.0001
audio/x-mpegurl 0.0001 0.0002 0.0002
image/bmp 0.0001 0.0001 0.0001
image/gif 0.0025 0.0024 0.0021
image/jp2 0.0001 0.0000 0.0000
image/jpeg 0.1310 0.1238 0.1281
image/png 0.0209 0.0183 0.0197
image/svg+xml 0.0005 0.0005 0.0004
image/tiff 0.0003 0.0005 0.0004
image/vnd.djvu 0.0006 0.0003 0.0002
image/vnd.dxf; format=ascii 0.0003 0.0003 0.0004
image/vnd.microsoft.icon 0.0002 0.0003 0.0001
image/webp 0.0009 0.0013 0.0012
message/rfc822 0.0013 0.0017 0.0016
text/calendar 0.0328 0.0496 0.0450
text/css 0.0003 0.0002 0.0003
text/csv 0.0026 0.0032 0.0028
text/html 78.3861 78.2290 79.2311
text/plain 0.0729 0.0742 0.0710
text/prs.lines.tag 0.0011 0.0010 0.0011
text/tab-separated-values 0.0004 0.0007 0.0005
text/troff 0.0003 0.0002 0.0004
text/turtle 0.0009 0.0010 0.0009
text/vtt 0.0005 0.0005 0.0004
text/x-c++src 0.0003 0.0004 0.0003
text/x-chdr 0.0010 0.0010 0.0007
text/x-csrc 0.0015 0.0015 0.0013
text/x-diff 0.0019 0.0019 0.0020
text/x-jsp 0.0001 0.0001 0.0001
text/x-log 0.0015 0.0013 0.0024
text/x-matlab 0.0015 0.0022 0.0019
text/x-patch 0.0002 0.0002 0.0001
text/x-perl 0.0031 0.0036 0.0034
text/x-php 0.0038 0.0050 0.0035
text/x-python 0.0003 0.0004 0.0003
text/x-vcalendar 0.0007 0.0010 0.0008
text/x-vcard 0.0033 0.0044 0.0039
video/mp4 0.0014 0.0015 0.0014
video/quicktime 0.0004 0.0005 0.0005
video/webm 0.0001 0.0001 0.0001
video/x-m4v 0.0001 0.0001 0.0001
video/x-matroska 0.0002 0.0002 0.0003
video/x-ms-wmv 0.0001 0.0001 0.0001