Statistics of Common Crawl Monthly Archives

Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives

View the Project on GitHub

MIME Types

The crawled content is dominated by HTML pages and contains only a small percentage of other document formats. The table shows the percentage of the top 100 media or MIME types of the latest monthly crawls.

crawl CC-MAIN-2018-51 CC-MAIN-2019-04 CC-MAIN-2019-09
mimetype
<other> 0.5516 0.5050 0.4326
application/atom+xml 0.0817 0.0968 0.0981
application/download 0.0014 0.0016 0.0018
application/epub+zip 0.0008 0.0009 0.0013
application/force-download 0.0033 0.0037 0.0040
application/javascript 0.0003 0.0003 0.0005
application/json 0.0094 0.0078 0.0129
application/jsonrequest 0.0000 0.0000 0.0000
application/ld+json 0.0001 0.0002 0.0003
application/marc 0.0000 0.0000 0.0001
application/msword 0.0042 0.0055 0.0083
application/octet-stream 0.0328 0.0418 0.0501
application/octetstream 0.0003 0.0004 0.0003
application/pdf 1.1398 0.9150 1.0602
application/pgp-encrypted 0.0000 0.0000 0.0000
application/pgp-signature 0.0005 0.0017 0.0009
application/postscript 0.0010 0.0006 0.0009
application/rdf+xml 0.0017 0.0013 0.0014
application/rss+xml 0.0795 0.0726 0.0837
application/rtf 0.0005 0.0007 0.0009
application/save-to-disk 0.0000 0.0000 0.0000
application/text 0.0003 0.0005 0.0010
application/unknown 0.0003 0.0004 0.0003
application/vnd.android.package-archive 0.0011 0.0011 0.0010
application/vnd.google-earth.kml+xml 0.0015 0.0011 0.0019
application/vnd.google-earth.kmz 0.0004 0.0003 0.0005
application/vnd.ms-excel 0.0014 0.0017 0.0029
application/vnd.ms-powerpoint 0.0004 0.0007 0.0010
application/vnd.ms-word 0.0001 0.0003 0.0003
application/vnd.ms-word.document.macroenabled.12 0.0013 0.0022 0.0008
application/vnd.openxmlformats-officedocument.presentationml.presentation 0.0003 0.0005 0.0007
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet 0.0009 0.0014 0.0018
application/vnd.openxmlformats-officedocument.wordprocessingml.document 0.0023 0.0032 0.0046
application/vnd.wap.xhtml+xml 0.0002 0.0002 0.0001
application/x-bibtex 0.0003 0.0006 0.0010
application/x-bittorrent 0.0004 0.0004 0.0004
application/x-debian-package 0.0000 0.0001 0.0003
application/x-download 0.0006 0.0010 0.0010
application/x-dvi 0.0000 0.0000 0.0001
application/x-endnote-refer 0.0004 0.0007 0.0012
application/x-gzip 0.0007 0.0015 0.0032
application/x-httpd-php 0.0007 0.0003 0.0008
application/x-java-archive 0.0000 0.0001 0.0002
application/x-java-jnlp-file 0.0001 0.0001 0.0001
application/x-javascript 0.0002 0.0002 0.0004
application/x-json 0.0000 0.0000 0.0000
application/x-mobipocket-ebook 0.0002 0.0001 0.0002
application/x-msdownload 0.0003 0.0005 0.0005
application/x-netcdf 0.0000 0.0001 0.0000
application/x-research-info-systems 0.0008 0.0013 0.0023
application/x-shockwave-flash 0.0015 0.0036 0.0041
application/x-tar 0.0002 0.0003 0.0003
application/x-tex 0.0001 0.0001 0.0001
application/x-troff-man 0.0001 0.0001 0.0002
application/x-xz 0.0006 0.0013 0.0007
application/x-zip-compressed 0.0005 0.0008 0.0013
application/xhtml+xml 0.0208 0.0196 0.0168
application/xml 0.0262 0.0231 0.0399
application/zip 0.0035 0.0056 0.0082
audio/mp3 0.0003 0.0003 0.0003
audio/mpeg 0.0056 0.0116 0.0173
audio/x-mpegurl 0.0014 0.0010 0.0017
audio/x-pn-realaudio 0.0001 0.0002 0.0002
audio/x-scpls 0.0002 0.0001 0.0002
audio/x-wav 0.0003 0.0004 0.0005
binary/octet-stream 0.0006 0.0006 0.0011
image/gif 0.0044 0.0067 0.0088
image/jp2 0.0000 0.0000 0.0001
image/jpeg 0.2732 0.2557 0.4850
image/jpg 0.0024 0.0032 0.0037
image/pjpeg 0.0003 0.0004 0.0004
image/png 0.0231 0.0308 0.0527
image/svg+xml 0.0013 0.0008 0.0013
image/tiff 0.0003 0.0006 0.0006
image/vnd.djvu 0.0004 0.0003 0.0006
image/webp 0.0005 0.0004 0.0004
message/rfc822 0.0002 0.0002 0.0002
text/calendar 0.0207 0.0257 0.0287
text/csv 0.0005 0.0007 0.0015
text/enriched 0.0001 0.0002 0.0003
text/html 97.5659 97.8144 97.3936
text/javascript 0.0003 0.0003 0.0003
text/pdf 0.0000 0.0000 0.0000
text/plain 0.0317 0.0295 0.0403
text/tab-separated-values 0.0001 0.0001 0.0002
text/turtle 0.0001 0.0001 0.0003
text/vnd.wap.wml 0.0002 0.0004 0.0003
text/x-bibtex 0.0003 0.0002 0.0004
text/x-c 0.0001 0.0001 0.0001
text/x-csrc 0.0001 0.0001 0.0001
text/x-diff 0.0002 0.0003 0.0003
text/x-patch 0.0000 0.0000 0.0000
text/x-perl 0.0001 0.0000 0.0001
text/x-vcalendar 0.0005 0.0007 0.0008
text/x-vcard 0.0019 0.0016 0.0020
text/xml 0.0791 0.0740 0.0850
unknown/unknown 0.0001 0.0001 0.0003
video/mp4 0.0018 0.0047 0.0068
video/webm 0.0033 0.0021 0.0036
video/x-ms-asf 0.0006 0.0004 0.0007
video/x-ms-wvx 0.0000 0.0000 0.0000