Statistics of Common Crawl Monthly Archives

Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives
Latest crawl: CC-MAIN-2024-10

View the Project on GitHub

MIME Types

The crawled content is dominated by HTML pages and contains only a small percentage of other document formats. The tables show the percentage of the top 100 media or MIME types of the latest monthly crawls.

While the first table is based the Content-Type HTTP header, the second uses the MIME type detected by Apache Tika based on the actual content. The underlying data including page counts is provided in mimetypes.csv resp. mimetypes_detected.csv.

crawl CC-MAIN-2023-40 CC-MAIN-2023-50 CC-MAIN-2024-10
mimetype
<other> 0.0700 0.0788 0.0748
application/atom+xml 0.0994 0.0960 0.0988
application/download 0.0030 0.0031 0.0032
application/epub+zip 0.0010 0.0013 0.0013
application/force-download 0.0063 0.0067 0.0062
application/javascript 0.0003 0.0004 0.0011
application/json 0.0209 0.0219 0.0226
application/ld+json 0.0014 0.0014 0.0016
application/marc 0.0003 0.0003 0.0004
application/msword 0.0032 0.0032 0.0032
application/octet-stream 0.0448 0.0467 0.0393
application/octetstream 0.0003 0.0002 0.0002
application/pdf 0.8661 1.0204 0.9381
application/pgp-encrypted 0.0001 0.0001 0.0001
application/pgp-signature 0.0023 0.0030 0.0027
application/postscript 0.0002 0.0002 0.0002
application/rdf+xml 0.0038 0.0037 0.0040
application/rss+xml 0.0537 0.0535 0.0553
application/rtf 0.0011 0.0011 0.0013
application/save-to-disk 0.0000 0.0000 0.0000
application/text 0.0006 0.0005 0.0002
application/unknown 0.0002 0.0003 0.0002
application/vnd.android.package-archive 0.0001 0.0001 0.0000
application/vnd.google-earth.kml+xml 0.0025 0.0012 0.0021
application/vnd.google-earth.kmz 0.0005 0.0006 0.0004
application/vnd.ms-excel 0.0018 0.0020 0.0020
application/vnd.ms-powerpoint 0.0003 0.0003 0.0003
application/vnd.ms-word 0.0003 0.0003 0.0003
application/vnd.oasis.opendocument.text 0.0008 0.0007 0.0007
application/vnd.openxmlformats-officedocument.presentationml.presentation 0.0002 0.0003 0.0003
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet 0.0014 0.0016 0.0015
application/vnd.openxmlformats-officedocument.wordprocessingml.document 0.0023 0.0025 0.0026
application/vnd.wap.xhtml+xml 0.0002 0.0005 0.0007
application/x-bibtex 0.0038 0.0038 0.0044
application/x-bittorrent 0.0002 0.0001 0.0001
application/x-bzip2 0.0001 0.0002 0.0002
application/x-debian-package 0.0000 0.0000 0.0000
application/x-download 0.0015 0.0014 0.0014
application/x-endnote-refer 0.0013 0.0012 0.0010
application/x-gzip 0.0004 0.0006 0.0006
application/x-httpd-php 0.0005 0.0004 0.0004
application/x-java-jnlp-file 0.0002 0.0001 0.0002
application/x-javascript 0.0001 0.0002 0.0001
application/x-json 0.0000 0.0000 0.0000
application/x-mobipocket-ebook 0.0002 0.0003 0.0003
application/x-msdownload 0.0002 0.0002 0.0002
application/x-netcdf 0.0014 0.0028 0.0004
application/x-rar-compressed 0.0001 0.0001 0.0000
application/x-research-info-systems 0.0075 0.0074 0.0078
application/x-shockwave-flash 0.0002 0.0002 0.0002
application/x-tar 0.0002 0.0001 0.0001
application/x-tex 0.0002 0.0002 0.0002
application/x-troff-man 0.0005 0.0007 0.0006
application/x-zip-compressed 0.0003 0.0004 0.0000
application/xhtml+xml 0.0165 0.0139 0.0090
application/xml 0.0231 0.0228 0.0224
application/zip 0.0023 0.0025 0.0002
audio/mp3 0.0001 0.0001 0.0000
audio/mpeg 0.0014 0.0013 0.0001
audio/x-mpegurl 0.0009 0.0007 0.0005
audio/x-pn-realaudio 0.0000 0.0000 0.0000
audio/x-scpls 0.0002 0.0001 0.0001
audio/x-wav 0.0001 0.0001 0.0000
binary/octet-stream 0.0007 0.0006 0.0006
image/gif 0.0035 0.0033 0.0008
image/jp2 0.0000 0.0000 0.0000
image/jpeg 0.1529 0.1566 0.0004
image/jpg 0.0016 0.0020 0.0000
image/pjpeg 0.0005 0.0003 0.0000
image/png 0.0297 0.0301 0.0002
image/svg+xml 0.0009 0.0010 0.0000
image/tiff 0.0003 0.0003 0.0000
image/vnd.djvu 0.0003 0.0003 NaN
image/webp 0.0023 0.0028 0.0000
message/rfc822 0.0002 0.0002 0.0002
text/calendar 0.0290 0.0300 0.0298
text/css 0.0005 0.0005 0.0007
text/csv 0.0035 0.0034 0.0033
text/directory 0.0002 0.0003 0.0002
text/enriched 0.0002 0.0001 0.0001
text/html 98.3908 98.2203 98.5183
text/javascript 0.0002 0.0002 0.0002
text/pdf 0.0000 0.0000 0.0000
text/plain 0.0575 0.0569 0.0601
text/prs.lines.tag 0.0018 0.0030 0.0025
text/tab-separated-values 0.0002 0.0002 0.0002
text/turtle 0.0013 0.0013 0.0016
text/vcard 0.0009 0.0010 0.0010
text/x-bibtex 0.0006 0.0006 0.0006
text/x-c 0.0001 0.0002 0.0002
text/x-csrc 0.0002 0.0003 0.0002
text/x-diff 0.0003 0.0003 0.0003
text/x-patch 0.0002 0.0003 0.0003
text/x-perl 0.0001 0.0001 0.0001
text/x-vcalendar 0.0004 0.0003 0.0004
text/x-vcard 0.0017 0.0020 0.0019
text/xml 0.0640 0.0651 0.0594
unknown/unknown 0.0000 0.0000 0.0000
video/mp4 0.0007 0.0007 0.0000
video/webm 0.0000 0.0001 0.0000
video/x-ms-asf 0.0001 0.0003 0.0002

crawl CC-MAIN-2023-40 CC-MAIN-2023-50 CC-MAIN-2024-10
mimetype_detected
<other> 0.0126 0.0167 0.0129
application/atom+xml 0.1018 0.0981 0.1006
application/epub+zip 0.0012 0.0016 0.0017
application/gpx+xml 0.0004 0.0006 0.0005
application/gzip 0.0014 0.0017 0.0000
application/javascript 0.0006 0.0007 0.0012
application/json 0.0202 0.0224 0.0232
application/marc 0.0010 0.0011 0.0010
application/mbox 0.0020 0.0018 0.0018
application/msword 0.0022 0.0024 0.0024
application/octet-stream 0.0321 0.0114 0.0107
application/pdf 0.8619 1.0242 0.9337
application/pgp-signature 0.0029 0.0026 0.0028
application/pkcs7-signature 0.0005 0.0006 0.0004
application/postscript 0.0002 0.0002 0.0001
application/rdf+xml 0.0084 0.0079 0.0085
application/rss+xml 0.0883 0.0884 0.0879
application/rtf 0.0013 0.0010 0.0011
application/text 0.0001 0.0001 0.0001
application/vnd.android.package-archive 0.0001 0.0001 0.0000
application/vnd.google-earth.kml+xml 0.0037 0.0026 0.0035
application/vnd.google-earth.kmz 0.0005 0.0006 0.0004
application/vnd.ms-excel 0.0014 0.0015 0.0016
application/vnd.ms-powerpoint 0.0003 0.0003 0.0003
application/vnd.oasis.opendocument.text 0.0009 0.0009 0.0009
application/vnd.openxmlformats-officedocument.presentationml.presentation 0.0002 0.0003 0.0003
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet 0.0014 0.0016 0.0015
application/vnd.openxmlformats-officedocument.wordprocessingml.document 0.0023 0.0025 0.0026
application/x-bibtex-text-file 0.0089 0.0079 0.0088
application/x-bittorrent 0.0002 0.0002 0.0002
application/x-bzip2 0.0000 0.0006 0.0000
application/x-debian-package 0.0000 0.0000 NaN
application/x-dosexec 0.0001 0.0001 0.0000
application/x-endnote-refer 0.0013 0.0013 0.0012
application/x-hdf 0.0006 0.0007 NaN
application/x-mobipocket-ebook 0.0003 0.0005 0.0005
application/x-ms-asx 0.0001 0.0003 0.0002
application/x-msdownload 0.0001 0.0001 0.0000
application/x-rar-compressed 0.0000 0.0000 0.0000
application/x-research-info-systems 0.0002 0.0001 0.0002
application/x-sh 0.0011 0.0010 0.0010
application/x-shockwave-flash 0.0002 0.0002 0.0003
application/x-stata-do 0.0005 0.0004 0.0003
application/x-tex 0.0003 0.0003 0.0003
application/x-tika-msoffice 0.0022 0.0021 0.0021
application/x-tika-ooxml 0.0026 0.0028 0.0028
application/x-wais-source 0.0002 0.0002 0.0002
application/x-xz 0.0003 0.0002 0.0000
application/xhtml+xml 9.9459 10.1468 9.3335
application/xml 0.0577 0.0615 0.0562
application/zip 0.0048 0.0052 0.0000
application/zlib 0.0001 0.0001 0.0001
application/zstd 0.0000 0.0000 NaN
audio/midi 0.0001 0.0001 0.0000
audio/mp4 0.0000 0.0000 NaN
audio/mpeg 0.0021 0.0019 0.0000
audio/vnd.wave 0.0003 0.0003 0.0000
audio/x-mpegurl 0.0001 0.0001 0.0000
image/bmp 0.0001 0.0001 0.0000
image/gif 0.0029 0.0027 0.0000
image/jpeg 0.1594 0.1635 0.0002
image/png 0.0301 0.0306 0.0000
image/svg+xml 0.0010 0.0011 0.0000
image/tiff 0.0003 0.0004 0.0000
image/vnd.djvu 0.0004 0.0005 NaN
image/vnd.dxf; format=ascii 0.0002 0.0003 NaN
image/webp 0.0024 0.0030 0.0000
message/rfc822 0.0020 0.0010 0.0007
text/asp 0.0000 0.0000 0.0000
text/aspdotnet 0.0000 0.0000 0.0000
text/calendar 0.0335 0.0351 0.0349
text/css 0.0005 0.0005 0.0007
text/csv 0.0031 0.0033 0.0032
text/html 88.4515 88.1096 89.2285
text/plain 0.1063 0.0919 0.0966
text/prs.lines.tag 0.0044 0.0064 0.0060
text/tab-separated-values 0.0002 0.0003 0.0002
text/troff 0.0005 0.0006 0.0006
text/turtle 0.0013 0.0013 0.0016
text/vtt 0.0007 0.0007 0.0008
text/x-c++src 0.0002 0.0002 0.0002
text/x-chdr 0.0006 0.0005 0.0005
text/x-csrc 0.0010 0.0012 0.0009
text/x-diff 0.0013 0.0011 0.0012
text/x-jsp 0.0001 0.0001 0.0001
text/x-log 0.0034 0.0017 0.0019
text/x-matlab 0.0021 0.0025 0.0022
text/x-perl 0.0020 0.0017 0.0016
text/x-php 0.0030 0.0031 0.0029
text/x-python 0.0003 0.0004 0.0004
text/x-vcalendar 0.0005 0.0004 0.0005
text/x-vcard 0.0032 0.0036 0.0035
text/x-web-markdown 0.0003 0.0005 0.0003
text/x-yaml 0.0002 0.0005 0.0003
video/mp4 0.0008 0.0008 0.0000
video/quicktime 0.0001 0.0001 NaN
video/webm 0.0000 0.0001 NaN
video/x-m4v 0.0000 0.0000 NaN
video/x-matroska 0.0000 0.0000 NaN