Statistics of Common Crawl Monthly Archives

Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives

View the Project on GitHub

MIME Types

The crawled content is dominated by HTML pages and contains only a small percentage of other document formats. The table shows the percentage of the top 100 media or MIME types of the latest monthly crawls.

crawl CC-MAIN-2018-43 CC-MAIN-2018-47 CC-MAIN-2018-51
mimetype
<other> 0.6520 0.6122 0.5519
application/atom+xml 0.0833 0.0948 0.0817
application/download 0.0014 0.0016 0.0014
application/epub+zip 0.0006 0.0008 0.0008
application/force-download 0.0029 0.0034 0.0033
application/javascript 0.0003 0.0005 0.0003
application/json 0.0099 0.0093 0.0094
application/jsonrequest 0.0000 0.0000 0.0000
application/ld+json 0.0002 0.0003 0.0001
application/marc 0.0000 0.0000 0.0000
application/msword 0.0048 0.0109 0.0042
application/octet-stream 0.0283 0.0458 0.0328
application/octetstream 0.0002 0.0002 0.0003
application/pdf 0.8036 0.9513 1.1398
application/pgp-encrypted 0.0000 0.0000 0.0000
application/pgp-signature 0.0005 0.0021 0.0005
application/postscript 0.0007 0.0011 0.0010
application/rdf+xml 0.0017 0.0023 0.0017
application/rss+xml 0.0721 0.0711 0.0795
application/save-to-disk 0.0000 nan 0.0000
application/text 0.0003 0.0003 0.0003
application/unknown 0.0003 0.0003 0.0003
application/vnd.android.package-archive 0.0008 0.0017 0.0011
application/vnd.google-earth.kml+xml 0.0010 0.0017 0.0015
application/vnd.google-earth.kmz 0.0003 0.0005 0.0004
application/vnd.ms-excel 0.0016 0.0033 0.0014
application/vnd.ms-powerpoint 0.0004 0.0011 0.0004
application/vnd.ms-word 0.0002 0.0001 0.0001
application/vnd.ms-word.document.macroenabled.12 0.0003 0.0031 0.0013
application/vnd.openxmlformats-officedocument.presentationml.presentation 0.0005 0.0003 0.0003
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet 0.0013 0.0010 0.0009
application/vnd.openxmlformats-officedocument.wordprocessingml.document 0.0038 0.0022 0.0023
application/vnd.wap.xhtml+xml 0.0003 0.0001 0.0002
application/x-bibtex 0.0004 0.0004 0.0003
application/x-bittorrent 0.0005 0.0005 0.0004
application/x-debian-package 0.0002 0.0001 0.0000
application/x-download 0.0007 0.0009 0.0006
application/x-dvi 0.0001 0.0001 0.0000
application/x-endnote-refer 0.0005 0.0006 0.0004
application/x-gzip 0.0010 0.0009 0.0007
application/x-httpd-php 0.0007 0.0008 0.0007
application/x-java-archive 0.0000 0.0000 0.0000
application/x-java-jnlp-file 0.0001 0.0001 0.0001
application/x-javascript 0.0002 0.0002 0.0002
application/x-json 0.0000 0.0000 0.0000
application/x-mobipocket-ebook 0.0001 0.0002 0.0002
application/x-msdownload 0.0004 0.0003 0.0003
application/x-netcdf 0.0000 0.0001 0.0000
application/x-research-info-systems 0.0008 0.0010 0.0008
application/x-shockwave-flash 0.0016 0.0021 0.0015
application/x-tar 0.0002 0.0003 0.0002
application/x-tex 0.0001 0.0001 0.0001
application/x-troff-man 0.0001 0.0001 0.0001
application/x-xz 0.0003 0.0017 0.0006
application/x-zip-compressed 0.0003 0.0007 0.0005
application/xhtml+xml 0.0210 0.0170 0.0208
application/xml 0.0279 0.0264 0.0262
application/zip 0.0035 0.0087 0.0035
audio/mp3 0.0001 0.0002 0.0003
audio/mpeg 0.0050 0.0065 0.0056
audio/x-mpegurl 0.0011 0.0019 0.0014
audio/x-pn-realaudio 0.0002 0.0001 0.0001
audio/x-scpls 0.0001 0.0002 0.0002
audio/x-wav 0.0004 0.0003 0.0003
binary/octet-stream 0.0004 0.0004 0.0006
image/gif 0.0052 0.0066 0.0044
image/jp2 0.0000 0.0000 0.0000
image/jpeg 0.2375 0.3043 0.2732
image/jpg 0.0021 0.0028 0.0024
image/pjpeg 0.0002 0.0003 0.0003
image/png 0.0205 0.0317 0.0231
image/svg+xml 0.0029 0.0024 0.0013
image/tiff 0.0004 0.0004 0.0003
image/vnd.djvu 0.0003 0.0006 0.0004
image/webp 0.0008 0.0004 0.0005
message/rfc822 0.0002 0.0004 0.0002
text/calendar 0.0176 0.0136 0.0207
text/csv 0.0005 0.0005 0.0005
text/enriched 0.0001 0.0001 0.0001
text/html 97.8543 97.6118 97.5659
text/javascript 0.0003 0.0003 0.0003
text/pdf 0.0000 0.0000 0.0000
text/plain 0.0263 0.0445 0.0317
text/tab-separated-values 0.0001 0.0001 0.0001
text/turtle 0.0001 0.0001 0.0001
text/vnd.wap.wml 0.0003 0.0005 0.0002
text/x-bibtex 0.0002 0.0002 0.0003
text/x-c 0.0001 0.0002 0.0001
text/x-csrc 0.0001 0.0001 0.0001
text/x-diff 0.0002 0.0002 0.0002
text/x-patch 0.0000 0.0000 0.0000
text/x-perl 0.0001 0.0003 0.0001
text/x-vcalendar 0.0004 0.0003 0.0005
text/x-vcard 0.0008 0.0010 0.0019
text/xml 0.0820 0.0731 0.0791
unknown/unknown 0.0002 0.0002 0.0001
video/mp4 0.0020 0.0023 0.0018
video/webm 0.0020 0.0028 0.0033
video/x-ms-asf 0.0004 0.0010 0.0006
video/x-ms-wmv 0.0005 0.0003 0.0002
video/x-ms-wvx 0.0000 0.0000 0.0000