Statistics of Common Crawl Monthly Archives

Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives
Latest crawl: CC-MAIN-2026-34

View the Project on GitHub

MIME Types

The crawled content is dominated by HTML pages and contains only a small percentage of other document formats. The tables show the percentage of the top 100 media or MIME types of the latest monthly crawls.

While the first table is based the Content-Type HTTP header, the second uses the MIME type detected by Apache Tika based on the actual content. The underlying data including page counts is provided in mimetypes.csv resp. mimetypes_detected.csv.

crawl CC-MAIN-2026-25 CC-MAIN-2026-30 CC-MAIN-2026-34
mimetype
<other> 0.0635 0.0630 0.0646
application/atom+xml 0.1595 0.1682 0.1804
application/calendar 0.0003 0.0002 0.0002
application/download 0.0045 0.0050 0.0047
application/epub+zip 0.0018 0.0018 0.0026
application/force-download 0.0113 0.0125 0.0124
application/gpx+xml 0.0009 0.0012 0.0011
application/ics 0.0004 0.0005 0.0005
application/javascript 0.0008 0.0013 0.0017
application/json 0.0256 0.0257 0.0272
application/ld+json 0.0012 0.0014 0.0013
application/marc 0.0003 0.0003 0.0003
application/msword 0.0028 0.0030 0.0029
application/octet-stream 0.0468 0.0454 0.0498
application/octetstream 0.0002 0.0001 0.0003
application/pdf 0.9791 0.9790 1.0160
application/pgp-encrypted 0.0002 0.0001 0.0002
application/pgp-signature 0.0019 0.0024 0.0023
application/postscript 0.0002 0.0002 0.0002
application/rdf+xml 0.0028 0.0029 0.0029
application/rss+xml 0.0428 0.0431 0.0453
application/rtf 0.0007 0.0008 0.0008
application/save-to-disk 0.0000 0.0000 0.0000
application/text 0.0002 0.0002 0.0002
application/unknown 0.0002 0.0003 0.0003
application/vnd.android.package-archive 0.0000 0.0000 0.0000
application/vnd.google-earth.kml+xml 0.0011 0.0011 0.0013
application/vnd.google-earth.kmz 0.0004 0.0005 0.0004
application/vnd.ms-excel 0.0015 0.0017 0.0013
application/vnd.ms-powerpoint 0.0002 0.0001 0.0001
application/vnd.ms-word 0.0001 0.0002 0.0002
application/vnd.oasis.opendocument.text 0.0009 0.0011 0.0011
application/vnd.openxmlformats-officedocument.presentationml.presentation 0.0002 0.0002 0.0003
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet 0.0018 0.0017 0.0018
application/vnd.openxmlformats-officedocument.wordprocessingml.document 0.0031 0.0034 0.0036
application/vnd.wap.xhtml+xml 0.0007 0.0006 0.0006
application/x-bibtex 0.0093 0.0097 0.0105
application/x-bittorrent 0.0003 0.0001 0.0001
application/x-debian-package 0.0000 0.0000 0.0000
application/x-download 0.0024 0.0025 0.0027
application/x-endnote-refer 0.0004 0.0005 0.0005
application/x-gzip 0.0002 0.0001 0.0002
application/x-httpd-php 0.0004 0.0002 0.0003
application/x-java-jnlp-file 0.0000 0.0000 0.0001
application/x-javascript 0.0001 0.0002 0.0003
application/x-json 0.0000 0.0000 0.0000
application/x-mobipocket-ebook 0.0005 0.0006 0.0010
application/x-msdownload 0.0003 0.0003 0.0003
application/x-netcdf 0.0002 0.0007 0.0007
application/x-research-info-systems 0.0115 0.0117 0.0127
application/x-shockwave-flash 0.0000 0.0000 0.0000
application/x-tar 0.0000 0.0001 0.0001
application/x-tex 0.0002 0.0001 0.0001
application/x-troff-man 0.0003 0.0004 0.0004
application/x-zip-compressed 0.0000 0.0000 0.0000
application/xhtml+xml 0.0098 0.0084 0.0088
application/xml 0.0256 0.0242 0.0253
application/zip 0.0002 0.0001 0.0002
audio/mpeg 0.0001 0.0001 0.0002
audio/x-mpegurl 0.0005 0.0005 0.0005
audio/x-scpls 0.0001 0.0001 0.0001
audio/x-wav 0.0000 0.0000 0.0000
binary/octet-stream 0.0005 0.0005 0.0005
image/gif 0.0001 0.0002 0.0002
image/jp2 0.0000 0.0000 0.0000
image/jpeg 0.0003 0.0003 0.0002
image/jpg 0.0000 0.0000 0.0000
image/pjpeg 0.0000 0.0000 0.0000
image/png 0.0000 0.0000 0.0001
image/svg+xml 0.0000 0.0000 0.0000
image/tiff 0.0000 0.0000 0.0000
image/vnd.djvu 0.0003 0.0005 0.0005
image/webp 0.0000 0.0000 0.0000
message/rfc822 0.0002 0.0002 0.0002
text/calendar 0.0352 0.0355 0.0383
text/css 0.0006 0.0008 0.0006
text/csv 0.0029 0.0029 0.0031
text/directory 0.0003 0.0003 0.0002
text/enriched 0.0000 0.0001 0.0001
text/html 98.4122 98.3618 98.2554
text/javascript 0.0004 0.0003 0.0003
text/markdown 0.0146 0.0225 0.0289
text/pdf 0.0000 0.0000 0.0000
text/plain 0.0526 0.0863 0.1181
text/prs.lines.tag 0.0024 0.0025 0.0033
text/tab-separated-values 0.0003 0.0003 0.0003
text/turtle 0.0011 0.0011 0.0012
text/vcard 0.0013 0.0013 0.0013
text/x-bibtex 0.0005 0.0006 0.0006
text/x-c 0.0001 0.0001 0.0001
text/x-csrc 0.0001 0.0002 0.0002
text/x-diff 0.0003 0.0003 0.0003
text/x-patch 0.0002 0.0003 0.0002
text/x-perl 0.0001 0.0001 0.0001
text/x-vcalendar 0.0004 0.0004 0.0004
text/x-vcard 0.0019 0.0019 0.0020
text/xml 0.0501 0.0484 0.0493
unknown/unknown 0.0000 0.0000 0.0000
video/mp4 0.0000 0.0000 0.0000
video/webm 0.0000 NaN NaN
video/x-ms-asf 0.0002 0.0002 0.0001

crawl CC-MAIN-2026-25 CC-MAIN-2026-30 CC-MAIN-2026-34
mimetype_detected
<other> 0.0123 0.0131 0.0157
application/atom+xml 0.1609 0.1696 0.1817
application/epub+zip 0.0021 0.0022 0.0030
application/gpx+xml 0.0009 0.0012 0.0011
application/gzip 0.0000 0.0000 0.0000
application/javascript 0.0012 0.0017 NaN
application/json 0.0257 0.0255 0.0266
application/marc 0.0007 0.0007 0.0008
application/mbox 0.0011 0.0012 0.0012
application/msword 0.0020 0.0022 0.0021
application/octet-stream 0.0127 0.0115 0.0116
application/pdf 0.9945 0.9933 1.0307
application/pgp-signature 0.0023 0.0024 0.0024
application/pkcs7-signature 0.0005 0.0007 0.0008
application/postscript 0.0001 0.0001 0.0001
application/rdf+xml 0.0059 0.0059 0.0063
application/rss+xml 0.0674 0.0667 0.0702
application/rtf 0.0011 0.0013 0.0014
application/text 0.0000 0.0000 0.0000
application/vnd.android.package-archive 0.0000 0.0000 0.0000
application/vnd.google-earth.kml+xml 0.0023 0.0024 0.0025
application/vnd.google-earth.kmz 0.0004 0.0005 0.0004
application/vnd.ms-excel 0.0009 0.0009 0.0009
application/vnd.ms-powerpoint 0.0002 0.0001 0.0001
application/vnd.oasis.opendocument.spreadsheet 0.0005 0.0005 0.0006
application/vnd.oasis.opendocument.text 0.0011 0.0012 0.0013
application/vnd.openxmlformats-officedocument.presentationml.presentation 0.0002 0.0002 0.0003
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet 0.0018 0.0017 0.0018
application/vnd.openxmlformats-officedocument.wordprocessingml.document 0.0031 0.0035 0.0036
application/x-bibtex-text-file 0.0141 0.0149 0.0160
application/x-bittorrent 0.0003 0.0002 0.0002
application/x-dosexec 0.0000 NaN 0.0000
application/x-endnote-refer 0.0018 0.0019 0.0019
application/x-mobipocket-ebook 0.0007 0.0008 0.0013
application/x-ms-asx 0.0002 0.0002 0.0001
application/x-msdownload 0.0000 0.0000 0.0000
application/x-pds 0.0006 0.0005 0.0013
application/x-rar-compressed 0.0000 NaN 0.0000
application/x-research-info-systems 0.0000 0.0000 0.0000
application/x-sh 0.0012 0.0010 0.0014
application/x-shockwave-flash 0.0000 0.0000 0.0000
application/x-stata-do 0.0004 0.0004 0.0005
application/x-tex 0.0004 0.0004 0.0003
application/x-tex-tfm 0.0003 0.0003 0.0002
application/x-tika-msoffice 0.0026 0.0025 0.0029
application/x-tika-ooxml 0.0023 0.0024 0.0024
application/x-wais-source 0.0001 0.0001 0.0003
application/x-xz NaN NaN 0.0000
application/xhtml+xml 7.9275 7.7386 7.4851
application/xml 0.0563 0.0542 0.0552
application/zip 0.0000 0.0000 0.0000
application/zlib 0.0002 0.0001 0.0001
audio/mpeg 0.0000 0.0000 0.0001
audio/vnd.wave 0.0000 0.0000 0.0000
audio/x-mpegurl 0.0000 0.0000 0.0000
image/gif 0.0000 0.0000 0.0000
image/jpeg 0.0002 0.0002 0.0001
image/png 0.0000 0.0000 0.0000
image/svg+xml 0.0000 0.0000 0.0000
image/tiff 0.0000 NaN NaN
image/vnd.djvu 0.0005 0.0009 0.0007
image/webp 0.0000 0.0000 0.0000
message/rfc822 0.0007 0.0009 0.0007
text/asp NaN 0.0000 0.0000
text/aspdotnet NaN NaN 0.0000
text/calendar 0.0433 0.0436 0.0474
text/css 0.0006 0.0007 0.0006
text/csv 0.0030 0.0029 0.0030
text/html 90.5115 90.6458 90.7932
text/markdown 0.0054 0.0093 0.0119
text/plain 0.0914 0.1220 0.1494
text/prs.lines.tag 0.0043 0.0047 0.0053
text/tab-separated-values 0.0003 0.0003 0.0003
text/troff 0.0004 0.0005 0.0006
text/turtle 0.0011 0.0011 0.0012
text/vtt 0.0008 0.0011 0.0012
text/x-c++src 0.0002 0.0002 0.0002
text/x-chdr 0.0008 0.0007 0.0012
text/x-csrc 0.0005 0.0006 0.0006
text/x-diff 0.0010 0.0011 0.0011
text/x-jsp 0.0001 0.0001 0.0001
text/x-log 0.0012 0.0012 0.0012
text/x-matlab 0.0022 0.0024 0.0023
text/x-perl 0.0009 0.0008 0.0008
text/x-php 0.0027 0.0045 0.0030
text/x-python 0.0004 0.0005 0.0005
text/x-robots 0.0004 0.0066 0.0139
text/x-vcalendar 0.0004 0.0004 0.0004
text/x-vcard 0.0038 0.0037 0.0039
text/x-web-markdown 0.0103 0.0141 0.0182
text/x-yaml 0.0004 0.0004 0.0005
video/mp4 0.0000 0.0000 0.0000
video/quicktime NaN NaN 0.0000
video/webm 0.0000 NaN NaN