Statistics of Common Crawl Monthly Archives

Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives

View the Project on GitHub

MIME Types

The crawled content is dominated by HTML pages and contains only a small percentage of other document formats. The table shows the percentage of the top 100 media or MIME types of the latest monthly crawls.

crawl CC-MAIN-2018-34 CC-MAIN-2018-39 CC-MAIN-2018-43
mimetype
<other> 0.6619 0.6530 0.6520
application/atom+xml 0.1002 0.0952 0.0833
application/download 0.0015 0.0008 0.0014
application/epub+zip 0.0005 0.0006 0.0006
application/force-download 0.0032 0.0016 0.0029
application/javascript 0.0009 0.0003 0.0003
application/json 0.0059 0.0080 0.0099
application/jsonrequest 0.0000 0.0000 0.0000
application/ld+json 0.0001 0.0001 0.0002
application/marc 0.0000 0.0000 0.0000
application/msword 0.0348 0.0023 0.0048
application/octet-stream 0.0310 0.0143 0.0283
application/octetstream 0.0002 0.0001 0.0002
application/pdf 0.4556 0.1419 0.8036
application/pgp-encrypted 0.0000 0.0000 0.0000
application/pgp-signature 0.0002 0.0002 0.0005
application/postscript 0.0008 0.0003 0.0007
application/rdf+xml 0.0019 0.0021 0.0017
application/rss+xml 0.0676 0.0667 0.0721
application/save-to-disk nan nan 0.0000
application/text 0.0003 0.0003 0.0003
application/unknown 0.0002 0.0001 0.0003
application/vnd.android.package-archive 0.0004 0.0004 0.0008
application/vnd.google-earth.kml+xml 0.0016 0.0013 0.0010
application/vnd.google-earth.kmz 0.0003 0.0002 0.0003
application/vnd.ms-excel 0.0082 0.0009 0.0016
application/vnd.ms-powerpoint 0.0046 0.0002 0.0004
application/vnd.ms-word 0.0002 0.0001 0.0002
application/vnd.ms-word.document.macroenabled.12 0.0008 0.0006 0.0003
application/vnd.openxmlformats-officedocument.presentationml.presentation 0.0022 0.0002 0.0005
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet 0.0040 0.0006 0.0013
application/vnd.openxmlformats-officedocument.wordprocessingml.document 0.0141 0.0018 0.0038
application/vnd.wap.xhtml+xml 0.0003 0.0002 0.0003
application/x-bibtex 0.0005 0.0004 0.0004
application/x-bittorrent 0.0010 0.0005 0.0005
application/x-debian-package 0.0006 0.0001 0.0002
application/x-download 0.0007 0.0002 0.0007
application/x-dvi 0.0000 0.0000 0.0001
application/x-endnote-refer 0.0005 0.0005 0.0005
application/x-gzip 0.0055 0.0005 0.0010
application/x-httpd-php 0.0003 0.0005 0.0007
application/x-java-archive 0.0001 0.0000 0.0000
application/x-java-jnlp-file 0.0001 0.0001 0.0001
application/x-javascript 0.0003 0.0005 0.0002
application/x-json 0.0000 0.0000 0.0000
application/x-mobipocket-ebook 0.0001 0.0001 0.0001
application/x-msdownload 0.0020 0.0002 0.0004
application/x-netcdf 0.0000 0.0000 0.0000
application/x-research-info-systems 0.0010 0.0008 0.0008
application/x-shockwave-flash 0.0054 0.0015 0.0016
application/x-tar 0.0008 0.0001 0.0002
application/x-tex 0.0000 0.0000 0.0001
application/x-troff-man 0.0001 0.0000 0.0001
application/x-xz 0.0001 0.0001 0.0003
application/x-zip-compressed 0.0038 0.0002 0.0003
application/xhtml+xml 0.0218 0.0255 0.0210
application/xml 0.0290 0.0223 0.0279
application/zip 0.0299 0.0021 0.0035
audio/mp3 0.0008 0.0001 0.0001
audio/mpeg 0.0860 0.0041 0.0050
audio/x-mpegurl 0.0009 0.0009 0.0011
audio/x-pn-realaudio 0.0021 0.0001 0.0002
audio/x-scpls 0.0001 0.0002 0.0001
audio/x-wav 0.0023 0.0002 0.0004
binary/octet-stream 0.0007 0.0002 0.0004
image/gif 0.0416 0.0053 0.0052
image/jp2 0.0000 0.0000 0.0000
image/jpeg 3.2095 0.2151 0.2375
image/jpg 0.0091 0.0014 0.0021
image/pjpeg 0.0008 0.0001 0.0002
image/png 0.2224 0.0203 0.0205
image/svg+xml 0.0029 0.0017 0.0029
image/tiff 0.0014 0.0002 0.0004
image/vnd.djvu 0.0001 0.0001 0.0003
image/webp 0.0004 0.0003 0.0008
message/rfc822 0.0001 0.0001 0.0002
text/calendar 0.0155 0.0160 0.0176
text/csv 0.0010 0.0004 0.0005
text/enriched 0.0001 0.0001 0.0001
text/html 94.7570 98.5925 97.8543
text/javascript 0.0003 0.0003 0.0003
text/pdf 0.0000 0.0000 0.0000
text/plain 0.0213 0.0188 0.0263
text/tab-separated-values 0.0001 0.0001 0.0001
text/turtle 0.0001 0.0001 0.0001
text/vnd.wap.wml 0.0003 0.0003 0.0003
text/x-bibtex 0.0001 0.0001 0.0002
text/x-c 0.0001 0.0001 0.0001
text/x-csrc 0.0000 0.0001 0.0001
text/x-diff 0.0001 0.0001 0.0002
text/x-patch 0.0000 0.0000 0.0000
text/x-perl 0.0001 0.0001 0.0001
text/x-vcalendar 0.0005 0.0004 0.0004
text/x-vcard 0.0008 0.0008 0.0008
text/xml 0.0860 0.0649 0.0820
unknown/unknown 0.0002 0.0002 0.0002
video/mp4 0.0220 0.0015 0.0020
video/webm 0.0009 0.0010 0.0020
video/x-ms-asf 0.0004 0.0003 0.0004
video/x-ms-wmv 0.0044 0.0002 0.0005
video/x-ms-wvx 0.0000 0.0000 0.0000