Statistics of Common Crawl Monthly Archives

Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives
Latest crawl: CC-MAIN-2024-18

View the Project on GitHub

MIME Types

The crawled content is dominated by HTML pages and contains only a small percentage of other document formats. The tables show the percentage of the top 100 media or MIME types of the latest monthly crawls.

While the first table is based the Content-Type HTTP header, the second uses the MIME type detected by Apache Tika based on the actual content. The underlying data including page counts is provided in mimetypes.csv resp. mimetypes_detected.csv.

crawl CC-MAIN-2023-50 CC-MAIN-2024-10 CC-MAIN-2024-18
mimetype
<other> 0.0788 0.0748 2.0973
application/atom+xml 0.0960 0.0988 0.0993
application/download 0.0031 0.0032 0.0034
application/epub+zip 0.0013 0.0013 0.0013
application/force-download 0.0067 0.0062 0.0068
application/javascript 0.0004 0.0011 0.0010
application/json 0.0219 0.0226 0.0254
application/ld+json 0.0014 0.0016 0.0016
application/marc 0.0003 0.0004 0.0005
application/msword 0.0032 0.0032 0.0033
application/octet-stream 0.0467 0.0393 0.0449
application/octetstream 0.0002 0.0002 0.0003
application/pdf 1.0204 0.9381 0.8970
application/pgp-encrypted 0.0001 0.0001 0.0001
application/pgp-signature 0.0030 0.0027 0.0028
application/postscript 0.0002 0.0002 0.0002
application/rdf+xml 0.0037 0.0040 0.0044
application/rss+xml 0.0535 0.0553 0.0556
application/rtf 0.0011 0.0013 0.0013
application/save-to-disk 0.0000 0.0000 0.0000
application/text 0.0005 0.0002 0.0003
application/unknown 0.0003 0.0002 0.0004
application/vnd.android.package-archive 0.0001 0.0000 0.0000
application/vnd.google-earth.kml+xml 0.0012 0.0021 0.0020
application/vnd.google-earth.kmz 0.0006 0.0004 0.0004
application/vnd.ms-excel 0.0020 0.0020 0.0021
application/vnd.ms-powerpoint 0.0003 0.0003 0.0003
application/vnd.ms-word 0.0003 0.0003 0.0003
application/vnd.oasis.opendocument.text 0.0007 0.0007 0.0007
application/vnd.openxmlformats-officedocument.presentationml.presentation 0.0003 0.0003 0.0002
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet 0.0016 0.0015 0.0016
application/vnd.openxmlformats-officedocument.wordprocessingml.document 0.0025 0.0026 0.0025
application/vnd.wap.xhtml+xml 0.0005 0.0007 0.0007
application/x-bibtex 0.0038 0.0044 0.0050
application/x-bittorrent 0.0001 0.0001 0.0002
application/x-bzip2 0.0002 0.0002 0.0002
application/x-debian-package 0.0000 0.0000 0.0000
application/x-download 0.0014 0.0014 0.0017
application/x-endnote-refer 0.0012 0.0010 0.0011
application/x-gzip 0.0006 0.0006 0.0002
application/x-httpd-php 0.0004 0.0004 0.0004
application/x-java-jnlp-file 0.0001 0.0002 0.0002
application/x-javascript 0.0002 0.0001 0.0001
application/x-json 0.0000 0.0000 0.0000
application/x-mobipocket-ebook 0.0003 0.0003 0.0003
application/x-msdownload 0.0002 0.0002 0.0002
application/x-netcdf 0.0028 0.0004 0.0001
application/x-rar-compressed 0.0001 0.0000 0.0000
application/x-research-info-systems 0.0074 0.0078 0.0100
application/x-shockwave-flash 0.0002 0.0002 0.0003
application/x-tar 0.0001 0.0001 0.0003
application/x-tex 0.0002 0.0002 0.0002
application/x-troff-man 0.0007 0.0006 0.0010
application/x-zip-compressed 0.0004 0.0000 0.0000
application/xhtml+xml 0.0139 0.0090 0.0092
application/xml 0.0228 0.0224 0.0228
application/zip 0.0025 0.0002 0.0001
audio/mp3 0.0001 0.0000 0.0000
audio/mpeg 0.0013 0.0001 0.0000
audio/x-mpegurl 0.0007 0.0005 0.0006
audio/x-pn-realaudio 0.0000 0.0000 0.0000
audio/x-scpls 0.0001 0.0001 0.0001
audio/x-wav 0.0001 0.0000 0.0000
binary/octet-stream 0.0006 0.0006 0.0007
image/gif 0.0033 0.0008 0.0007
image/jp2 0.0000 0.0000 0.0000
image/jpeg 0.1566 0.0004 0.0003
image/jpg 0.0020 0.0000 0.0000
image/pjpeg 0.0003 0.0000 0.0000
image/png 0.0301 0.0002 0.0003
image/svg+xml 0.0010 0.0000 0.0000
image/tiff 0.0003 0.0000 0.0000
image/vnd.djvu 0.0003 NaN 0.0000
image/webp 0.0028 0.0000 0.0000
message/rfc822 0.0002 0.0002 0.0003
text/calendar 0.0300 0.0298 0.0298
text/css 0.0005 0.0007 0.0007
text/csv 0.0034 0.0033 0.0032
text/directory 0.0003 0.0002 0.0002
text/enriched 0.0001 0.0001 0.0001
text/html 98.2203 98.5183 96.5107
text/javascript 0.0002 0.0002 0.0002
text/pdf 0.0000 0.0000 0.0000
text/plain 0.0569 0.0601 0.0666
text/prs.lines.tag 0.0030 0.0025 0.0036
text/tab-separated-values 0.0002 0.0002 0.0002
text/turtle 0.0013 0.0016 0.0016
text/vcard 0.0010 0.0010 0.0010
text/x-bibtex 0.0006 0.0006 0.0007
text/x-c 0.0002 0.0002 0.0001
text/x-csrc 0.0003 0.0002 0.0003
text/x-diff 0.0003 0.0003 0.0005
text/x-patch 0.0003 0.0003 0.0003
text/x-perl 0.0001 0.0001 0.0001
text/x-vcalendar 0.0003 0.0004 0.0004
text/x-vcard 0.0020 0.0019 0.0021
text/xml 0.0651 0.0594 0.0627
unknown/unknown 0.0000 0.0000 0.0000
video/mp4 0.0007 0.0000 0.0000
video/webm 0.0001 0.0000 0.0000
video/x-ms-asf 0.0003 0.0002 0.0001

crawl CC-MAIN-2023-50 CC-MAIN-2024-10 CC-MAIN-2024-18
mimetype_detected
<other> 0.0167 0.0129 0.0143
application/atom+xml 0.0981 0.1006 0.1009
application/epub+zip 0.0016 0.0017 0.0017
application/gpx+xml 0.0006 0.0005 0.0005
application/gzip 0.0017 0.0000 0.0000
application/javascript 0.0007 0.0012 0.0012
application/json 0.0224 0.0232 0.0260
application/marc 0.0011 0.0010 0.0012
application/mbox 0.0018 0.0018 0.0020
application/msword 0.0024 0.0024 0.0023
application/octet-stream 0.0114 0.0107 0.0115
application/pdf 1.0242 0.9337 0.9146
application/pgp-signature 0.0026 0.0028 0.0028
application/pkcs7-signature 0.0006 0.0004 0.0004
application/postscript 0.0002 0.0001 0.0001
application/rdf+xml 0.0079 0.0085 0.0091
application/rss+xml 0.0884 0.0879 0.0878
application/rtf 0.0010 0.0011 0.0011
application/text 0.0001 0.0001 0.0000
application/vnd.android.package-archive 0.0001 0.0000 0.0000
application/vnd.google-earth.kml+xml 0.0026 0.0035 0.0035
application/vnd.google-earth.kmz 0.0006 0.0004 0.0004
application/vnd.ms-excel 0.0015 0.0016 0.0017
application/vnd.ms-powerpoint 0.0003 0.0003 0.0003
application/vnd.oasis.opendocument.text 0.0009 0.0009 0.0008
application/vnd.openxmlformats-officedocument.presentationml.presentation 0.0003 0.0003 0.0002
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet 0.0016 0.0015 0.0016
application/vnd.openxmlformats-officedocument.wordprocessingml.document 0.0025 0.0026 0.0025
application/x-bibtex-text-file 0.0079 0.0088 0.0103
application/x-bittorrent 0.0002 0.0002 0.0002
application/x-bzip2 0.0006 0.0000 NaN
application/x-debian-package 0.0000 NaN 0.0000
application/x-dosexec 0.0001 0.0000 0.0000
application/x-endnote-refer 0.0013 0.0012 0.0016
application/x-hdf 0.0007 NaN NaN
application/x-mobipocket-ebook 0.0005 0.0005 0.0005
application/x-ms-asx 0.0003 0.0002 0.0001
application/x-msdownload 0.0001 0.0000 0.0000
application/x-rar-compressed 0.0000 0.0000 0.0000
application/x-research-info-systems 0.0001 0.0002 0.0002
application/x-sh 0.0010 0.0010 0.0010
application/x-shockwave-flash 0.0002 0.0003 0.0003
application/x-stata-do 0.0004 0.0003 0.0005
application/x-tex 0.0003 0.0003 0.0003
application/x-tika-msoffice 0.0021 0.0021 0.0024
application/x-tika-ooxml 0.0028 0.0028 0.0029
application/x-wais-source 0.0002 0.0002 0.0002
application/x-xz 0.0002 0.0000 NaN
application/xhtml+xml 10.1468 9.3335 9.3876
application/xml 0.0615 0.0562 0.0606
application/zip 0.0052 0.0000 0.0000
application/zlib 0.0001 0.0001 0.0001
application/zstd 0.0000 NaN NaN
audio/midi 0.0001 0.0000 NaN
audio/mp4 0.0000 NaN NaN
audio/mpeg 0.0019 0.0000 0.0000
audio/vnd.wave 0.0003 0.0000 0.0000
audio/x-mpegurl 0.0001 0.0000 0.0000
image/bmp 0.0001 0.0000 NaN
image/gif 0.0027 0.0000 0.0000
image/jpeg 0.1635 0.0002 0.0002
image/png 0.0306 0.0000 0.0000
image/svg+xml 0.0011 0.0000 0.0000
image/tiff 0.0004 0.0000 0.0000
image/vnd.djvu 0.0005 NaN 0.0000
image/vnd.dxf; format=ascii 0.0003 NaN NaN
image/webp 0.0030 0.0000 0.0000
message/rfc822 0.0010 0.0007 0.0005
text/asp 0.0000 0.0000 0.0000
text/aspdotnet 0.0000 0.0000 0.0000
text/calendar 0.0351 0.0349 0.0352
text/css 0.0005 0.0007 0.0007
text/csv 0.0033 0.0032 0.0032
text/html 88.1096 89.2285 89.1642
text/plain 0.0919 0.0966 0.1082
text/prs.lines.tag 0.0064 0.0060 0.0084
text/tab-separated-values 0.0003 0.0002 0.0002
text/troff 0.0006 0.0006 0.0009
text/turtle 0.0013 0.0016 0.0016
text/vtt 0.0007 0.0008 0.0008
text/x-c++src 0.0002 0.0002 0.0002
text/x-chdr 0.0005 0.0005 0.0005
text/x-csrc 0.0012 0.0009 0.0011
text/x-diff 0.0011 0.0012 0.0013
text/x-jsp 0.0001 0.0001 0.0001
text/x-log 0.0017 0.0019 0.0021
text/x-matlab 0.0025 0.0022 0.0022
text/x-perl 0.0017 0.0016 0.0017
text/x-php 0.0031 0.0029 0.0035
text/x-python 0.0004 0.0004 0.0005
text/x-vcalendar 0.0004 0.0005 0.0004
text/x-vcard 0.0036 0.0035 0.0037
text/x-web-markdown 0.0005 0.0003 0.0004
text/x-yaml 0.0005 0.0003 0.0004
video/mp4 0.0008 0.0000 0.0000
video/quicktime 0.0001 NaN NaN
video/webm 0.0001 NaN 0.0000
video/x-m4v 0.0000 NaN NaN
video/x-matroska 0.0000 NaN NaN