Statistics of Common Crawl Monthly Archives

Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives

View the Project on GitHub

MIME Types

The crawled content is dominated by HTML pages and contains only a small percentage of other document formats. The tables show the percentage of the top 100 media or MIME types of the latest monthly crawls.

While the first table is based the Content-Type HTTP header, the second uses the MIME type detected by Apache Tika based on the actual content.

crawl CC-MAIN-2019-47 CC-MAIN-2019-51 CC-MAIN-2020-05
mimetype
<other> 0.7747 0.6482 0.7261
application/atom+xml 0.1217 0.1285 0.1232
application/download 0.0019 0.0026 0.0021
application/epub+zip 0.0018 0.0014 0.0010
application/force-download 0.0051 0.0046 0.0044
application/javascript 0.0003 0.0003 0.0004
application/json 0.0227 0.0229 0.0189
application/ld+json 0.0003 0.0004 0.0003
application/marc 0.0004 0.0005 0.0004
application/msword 0.0032 0.0029 0.0020
application/octet-stream 0.0456 0.0454 0.0352
application/octetstream 0.0003 0.0002 0.0001
application/pdf 0.4531 0.1212 0.0792
application/pgp-encrypted 0.0001 0.0000 0.0000
application/pgp-signature 0.0013 0.0009 0.0009
application/postscript 0.0006 0.0002 0.0001
application/rdf+xml 0.0026 0.0028 0.0023
application/rss+xml 0.0737 0.0779 0.0726
application/rtf 0.0011 0.0010 0.0009
application/save-to-disk 0.0000 0.0000 0.0000
application/text 0.0029 0.0030 0.0027
application/unknown 0.0003 0.0002 0.0003
application/vnd.android.package-archive 0.0011 0.0012 0.0008
application/vnd.google-earth.kml+xml 0.0016 0.0017 0.0016
application/vnd.google-earth.kmz 0.0003 0.0003 0.0003
application/vnd.ms-excel 0.0013 0.0014 0.0011
application/vnd.ms-powerpoint 0.0016 0.0009 0.0007
application/vnd.ms-word 0.0007 0.0007 0.0006
application/vnd.ms-word.document.macroenabled.12 0.0000 0.0000 0.0000
application/vnd.oasis.opendocument.text 0.0007 0.0007 0.0006
application/vnd.openxmlformats-officedocument.presentationml.presentation 0.0002 0.0003 0.0002
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet 0.0012 0.0013 0.0010
application/vnd.openxmlformats-officedocument.wordprocessingml.document 0.0023 0.0023 0.0018
application/vnd.wap.xhtml+xml 0.0001 0.0001 0.0001
application/x-bibtex 0.0018 0.0020 0.0017
application/x-bittorrent 0.0003 0.0003 0.0003
application/x-debian-package 0.0002 0.0000 0.0001
application/x-download 0.0011 0.0013 0.0010
application/x-endnote-refer 0.0036 0.0037 0.0032
application/x-gzip 0.0009 0.0009 0.0008
application/x-httpd-php 0.0007 0.0006 0.0006
application/x-java-jnlp-file 0.0003 0.0003 0.0003
application/x-javascript 0.0003 0.0006 0.0004
application/x-json 0.0000 0.0000 0.0000
application/x-mobipocket-ebook 0.0002 0.0002 0.0001
application/x-msdownload 0.0003 0.0003 0.0003
application/x-netcdf 0.0002 0.0002 0.0000
application/x-research-info-systems 0.0063 0.0069 0.0057
application/x-shockwave-flash 0.0005 0.0006 0.0005
application/x-tar 0.0002 0.0002 0.0002
application/x-tex 0.0001 0.0001 0.0001
application/x-troff-man 0.0003 0.0006 0.0003
application/x-xz 0.0000 0.0000 0.0000
application/x-zip-compressed 0.0007 0.0007 0.0005
application/xhtml+xml 0.0401 0.0451 0.0374
application/xml 0.0265 0.0292 0.0290
application/zip 0.0040 0.0040 0.0035
audio/mp3 0.0002 0.0003 0.0002
audio/mp4 0.0008 0.0007 0.0004
audio/mpeg 0.0044 0.0042 0.0029
audio/x-mpegurl 0.0009 0.0011 0.0010
audio/x-pn-realaudio 0.0001 0.0001 0.0001
audio/x-scpls 0.0001 0.0001 0.0001
audio/x-wav 0.0008 0.0007 0.0006
binary/octet-stream 0.0016 0.0006 0.0006
image/gif 0.0027 0.0027 0.0024
image/jp2 0.0002 0.0002 0.0002
image/jpeg 0.1410 0.1279 0.0963
image/jpg 0.0027 0.0027 0.0020
image/pjpeg 0.0005 0.0007 0.0006
image/png 0.0219 0.0190 0.0142
image/svg+xml 0.0006 0.0004 0.0003
image/tiff 0.0004 0.0004 0.0003
image/vnd.djvu 0.0003 0.0003 0.0003
image/webp 0.0007 0.0007 0.0005
message/rfc822 0.0004 0.0005 0.0004
text/calendar 0.0356 0.0422 0.0342
text/csv 0.0028 0.0028 0.0026
text/enriched 0.0013 0.0012 0.0010
text/html 98.0216 98.4623 98.5013
text/javascript 0.0005 0.0006 0.0006
text/pdf 0.0000 0.0000 0.0000
text/plain 0.0536 0.0514 0.0720
text/tab-separated-values 0.0006 0.0007 0.0005
text/turtle 0.0007 0.0009 0.0008
text/vcard 0.0009 0.0010 0.0009
text/vnd.wap.wml 0.0002 0.0002 0.0002
text/x-bibtex 0.0002 0.0003 0.0004
text/x-c 0.0001 0.0001 0.0001
text/x-csrc 0.0008 0.0008 0.0007
text/x-diff 0.0003 0.0003 0.0002
text/x-patch 0.0004 0.0004 0.0004
text/x-perl 0.0001 0.0002 0.0001
text/x-vcalendar 0.0006 0.0007 0.0005
text/x-vcard 0.0019 0.0022 0.0021
text/xml 0.0819 0.0926 0.0880
unknown/unknown 0.0006 0.0006 0.0006
video/mp4 0.0014 0.0014 0.0010
video/webm 0.0001 0.0001 0.0001
video/x-ms-asf 0.0001 0.0002 0.0004

crawl CC-MAIN-2019-47 CC-MAIN-2019-51 CC-MAIN-2020-05
mimetype_detected
<other> 0.0135 0.0143 0.0121
application/atom+xml 0.1259 0.1332 0.1280
application/epub+zip 0.0022 0.0017 0.0013
application/force-download 0.0001 0.0001 0.0002
application/gzip 0.0027 0.0025 0.0015
application/javascript 0.0011 0.0014 0.0012
application/json 0.0245 0.0244 0.0197
application/marc 0.0002 0.0003 0.0002
application/mbox 0.0024 0.0024 0.0022
application/msword 0.0027 0.0023 0.0015
application/octet-stream 0.0127 0.0134 0.0105
application/pdf 0.4712 0.1325 0.0876
application/pgp-signature 0.0015 0.0009 0.0010
application/pls+xml 0.0001 0.0001 0.0001
application/postscript 0.0005 0.0002 0.0001
application/rdf+xml 0.0093 0.0099 0.0096
application/rss+xml 0.1322 0.1417 0.1307
application/rtf 0.0027 0.0026 0.0022
application/text 0.0005 0.0005 0.0005
application/vnd.android.package-archive 0.0015 0.0015 0.0010
application/vnd.google-earth.kml+xml 0.0021 0.0022 0.0020
application/vnd.google-earth.kmz 0.0004 0.0003 0.0004
application/vnd.ms-excel 0.0012 0.0012 0.0008
application/vnd.ms-powerpoint 0.0016 0.0009 0.0007
application/vnd.oasis.opendocument.text 0.0007 0.0008 0.0006
application/vnd.openxmlformats-officedocument.presentationml.presentation 0.0002 0.0003 0.0002
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet 0.0012 0.0012 0.0010
application/vnd.openxmlformats-officedocument.wordprocessingml.document 0.0024 0.0023 0.0018
application/x-bibtex-text-file 0.0049 0.0052 0.0051
application/x-bittorrent 0.0003 0.0004 0.0003
application/x-bzip2 0.0007 0.0009 0.0003
application/x-debian-package 0.0002 0.0002 0.0002
application/x-dosexec 0.0005 0.0005 0.0004
application/x-endnote-refer 0.0019 0.0019 0.0017
application/x-mobipocket-ebook 0.0004 0.0003 0.0003
application/x-ms-asx 0.0001 0.0002 0.0003
application/x-msdownload 0.0001 0.0001 0.0001
application/x-rar-compressed 0.0005 0.0006 0.0004
application/x-research-info-systems 0.0002 0.0003 0.0002
application/x-sh 0.0013 0.0008 0.0009
application/x-shockwave-flash 0.0005 0.0006 0.0005
application/x-stata-do 0.0002 0.0003 0.0004
application/x-tika-msoffice 0.0029 0.0031 0.0025
application/x-tika-ooxml 0.0019 0.0020 0.0017
application/x-wais-source 0.0002 0.0002 0.0002
application/x-xz 0.0002 0.0003 0.0002
application/xhtml+xml 21.4491 21.6764 21.0508
application/xml 0.0661 0.0767 0.0794
application/zip 0.0086 0.0081 0.0065
application/zlib 0.0004 0.0005 0.0003
audio/midi 0.0001 0.0001 0.0001
audio/mp4 0.0014 0.0013 0.0009
audio/mpeg 0.0057 0.0061 0.0043
audio/vnd.wave 0.0009 0.0009 0.0008
audio/vorbis 0.0001 0.0001 0.0001
audio/x-mpegurl 0.0001 0.0001 0.0001
image/bmp 0.0001 0.0002 0.0001
image/gif 0.0028 0.0028 0.0024
image/jp2 0.0002 0.0002 0.0002
image/jpeg 0.1518 0.1378 0.1039
image/png 0.0225 0.0200 0.0152
image/svg+xml 0.0007 0.0005 0.0004
image/tiff 0.0021 0.0011 0.0005
image/vnd.djvu 0.0005 0.0005 0.0005
image/vnd.dxf; format=ascii 0.0003 0.0005 0.0003
image/vnd.microsoft.icon 0.0005 0.0003 0.0002
image/webp 0.0009 0.0009 0.0006
message/rfc822 0.0012 0.0014 0.0013
text/calendar 0.0370 0.0438 0.0357
text/css 0.0004 0.0003 0.0003
text/csv 0.0026 0.0027 0.0024
text/html 77.3011 77.3944 78.1387
text/plain 0.0799 0.0804 0.0930
text/prs.lines.tag 0.0016 0.0013 0.0015
text/tab-separated-values 0.0006 0.0007 0.0005
text/troff 0.0004 0.0006 0.0003
text/turtle 0.0007 0.0009 0.0008
text/x-c++src 0.0004 0.0003 0.0003
text/x-chdr 0.0011 0.0013 0.0010
text/x-csrc 0.0018 0.0019 0.0017
text/x-diff 0.0021 0.0022 0.0019
text/x-jsp 0.0002 0.0002 0.0002
text/x-log 0.0029 0.0027 0.0024
text/x-matlab 0.0021 0.0022 0.0018
text/x-patch 0.0002 0.0002 0.0001
text/x-perl 0.0031 0.0035 0.0039
text/x-php 0.0034 0.0033 0.0035
text/x-python 0.0003 0.0003 0.0003
text/x-vcalendar 0.0007 0.0009 0.0006
text/x-vcard 0.0037 0.0041 0.0038
video/mp4 0.0016 0.0016 0.0012
video/quicktime 0.0004 0.0004 0.0002
video/webm 0.0001 0.0001 0.0001
video/x-m4v 0.0006 0.0005 0.0001
video/x-matroska 0.0002 0.0002 0.0002
video/x-ms-wmv 0.0001 0.0001 0.0001