Statistics of Common Crawl Monthly Archives

Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives
Latest crawl: CC-MAIN-2026-39

View the Project on GitHub

MIME Types

The crawled content is dominated by HTML pages and contains only a small percentage of other document formats. The tables show the percentage of the top 100 media or MIME types of the latest monthly crawls.

While the first table is based the Content-Type HTTP header, the second uses the MIME type detected by Apache Tika based on the actual content. The underlying data including page counts is provided in mimetypes.csv resp. mimetypes_detected.csv.

crawl CC-MAIN-2026-30 CC-MAIN-2026-34 CC-MAIN-2026-39
mimetype
<other> 0.0630 0.0646 0.0706
application/atom+xml 0.1682 0.1804 0.1570
application/calendar 0.0002 0.0002 0.0002
application/download 0.0050 0.0047 0.0043
application/epub+zip 0.0018 0.0026 0.0019
application/force-download 0.0125 0.0124 0.0118
application/gpx+xml 0.0012 0.0011 0.0011
application/ics 0.0005 0.0005 0.0004
application/javascript 0.0013 0.0017 0.0012
application/json 0.0257 0.0272 0.0298
application/ld+json 0.0014 0.0013 0.0012
application/marc 0.0003 0.0003 0.0003
application/msword 0.0030 0.0029 0.0028
application/octet-stream 0.0454 0.0498 0.0493
application/octetstream 0.0001 0.0003 0.0003
application/pdf 0.9790 1.0160 0.9497
application/pgp-encrypted 0.0001 0.0002 0.0001
application/pgp-signature 0.0024 0.0023 0.0028
application/postscript 0.0002 0.0002 0.0002
application/rdf+xml 0.0029 0.0029 0.0029
application/rss+xml 0.0431 0.0453 0.0435
application/rtf 0.0008 0.0008 0.0007
application/save-to-disk 0.0000 0.0000 0.0000
application/text 0.0002 0.0002 0.0001
application/unknown 0.0003 0.0003 0.0002
application/vnd.android.package-archive 0.0000 0.0000 0.0000
application/vnd.google-earth.kml+xml 0.0011 0.0013 0.0013
application/vnd.google-earth.kmz 0.0005 0.0004 0.0003
application/vnd.ms-excel 0.0017 0.0013 0.0013
application/vnd.ms-powerpoint 0.0001 0.0001 0.0001
application/vnd.ms-word 0.0002 0.0002 0.0002
application/vnd.oasis.opendocument.text 0.0011 0.0011 0.0009
application/vnd.openxmlformats-officedocument.presentationml.presentation 0.0002 0.0003 0.0003
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet 0.0017 0.0018 0.0017
application/vnd.openxmlformats-officedocument.wordprocessingml.document 0.0034 0.0036 0.0032
application/vnd.wap.xhtml+xml 0.0006 0.0006 0.0007
application/x-bibtex 0.0097 0.0105 0.0107
application/x-bittorrent 0.0001 0.0001 0.0003
application/x-debian-package 0.0000 0.0000 0.0000
application/x-download 0.0025 0.0027 0.0033
application/x-endnote-refer 0.0005 0.0005 0.0005
application/x-gzip 0.0001 0.0002 0.0002
application/x-httpd-php 0.0002 0.0003 0.0003
application/x-java-jnlp-file 0.0000 0.0001 0.0000
application/x-javascript 0.0002 0.0003 0.0002
application/x-json 0.0000 0.0000 0.0000
application/x-mobipocket-ebook 0.0006 0.0010 0.0006
application/x-msdownload 0.0003 0.0003 0.0003
application/x-netcdf 0.0007 0.0007 0.0001
application/x-research-info-systems 0.0117 0.0127 0.0128
application/x-shockwave-flash 0.0000 0.0000 0.0000
application/x-tar 0.0001 0.0001 0.0001
application/x-tex 0.0001 0.0001 0.0002
application/x-troff-man 0.0004 0.0004 0.0004
application/x-zip-compressed 0.0000 0.0000 0.0000
application/xhtml+xml 0.0084 0.0088 0.0082
application/xml 0.0242 0.0253 0.0247
application/zip 0.0001 0.0002 0.0002
audio/mpeg 0.0001 0.0002 0.0000
audio/x-mpegurl 0.0005 0.0005 0.0005
audio/x-scpls 0.0001 0.0001 0.0001
audio/x-wav 0.0000 0.0000 0.0000
binary/octet-stream 0.0005 0.0005 0.0005
image/gif 0.0002 0.0002 0.0001
image/jp2 0.0000 0.0000 0.0000
image/jpeg 0.0003 0.0002 0.0003
image/jpg 0.0000 0.0000 0.0000
image/pjpeg 0.0000 0.0000 0.0000
image/png 0.0000 0.0001 0.0000
image/svg+xml 0.0000 0.0000 0.0000
image/tiff 0.0000 0.0000 0.0000
image/vnd.djvu 0.0005 0.0005 0.0004
image/webp 0.0000 0.0000 0.0000
message/rfc822 0.0002 0.0002 0.0003
text/calendar 0.0355 0.0383 0.0376
text/css 0.0008 0.0006 0.0006
text/csv 0.0029 0.0031 0.0028
text/directory 0.0003 0.0002 0.0002
text/enriched 0.0001 0.0001 0.0000
text/html 98.3618 98.2554 98.3804
text/javascript 0.0003 0.0003 0.0003
text/markdown 0.0225 0.0289 0.0288
text/pdf 0.0000 0.0000 0.0000
text/plain 0.0863 0.1181 0.0861
text/prs.lines.tag 0.0025 0.0033 0.0037
text/tab-separated-values 0.0003 0.0003 0.0003
text/turtle 0.0011 0.0012 0.0010
text/vcard 0.0013 0.0013 0.0014
text/x-bibtex 0.0006 0.0006 0.0005
text/x-c 0.0001 0.0001 0.0001
text/x-csrc 0.0002 0.0002 0.0001
text/x-diff 0.0003 0.0003 0.0003
text/x-patch 0.0003 0.0002 0.0002
text/x-perl 0.0001 0.0001 0.0000
text/x-vcalendar 0.0004 0.0004 0.0004
text/x-vcard 0.0019 0.0020 0.0020
text/xml 0.0484 0.0493 0.0449
unknown/unknown 0.0000 0.0000 0.0000
video/mp4 0.0000 0.0000 0.0000
video/x-ms-asf 0.0002 0.0001 0.0002

crawl CC-MAIN-2026-30 CC-MAIN-2026-34 CC-MAIN-2026-39
mimetype_detected
<other> 0.0131 0.0157 0.0145
application/atom+xml 0.1696 0.1817 0.1583
application/epub+zip 0.0022 0.0030 0.0023
application/gpx+xml 0.0012 0.0011 0.0011
application/gzip 0.0000 0.0000 0.0000
application/javascript 0.0017 NaN NaN
application/json 0.0255 0.0266 0.0260
application/marc 0.0007 0.0008 0.0006
application/mbox 0.0012 0.0012 0.0011
application/msword 0.0022 0.0021 0.0021
application/octet-stream 0.0115 0.0116 0.0154
application/pdf 0.9933 1.0307 0.9647
application/pgp-signature 0.0024 0.0024 0.0025
application/pkcs7-signature 0.0007 0.0008 0.0007
application/postscript 0.0001 0.0001 0.0001
application/rdf+xml 0.0059 0.0063 0.0060
application/rss+xml 0.0667 0.0702 0.0661
application/rtf 0.0013 0.0014 0.0012
application/text 0.0000 0.0000 0.0000
application/vnd.android.package-archive 0.0000 0.0000 0.0000
application/vnd.google-earth.kml+xml 0.0024 0.0025 0.0027
application/vnd.google-earth.kmz 0.0005 0.0004 0.0004
application/vnd.ms-excel 0.0009 0.0009 0.0008
application/vnd.ms-powerpoint 0.0001 0.0001 0.0001
application/vnd.oasis.opendocument.spreadsheet 0.0005 0.0006 0.0005
application/vnd.oasis.opendocument.text 0.0012 0.0013 0.0010
application/vnd.openxmlformats-officedocument.presentationml.presentation 0.0002 0.0003 0.0003
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet 0.0017 0.0018 0.0017
application/vnd.openxmlformats-officedocument.wordprocessingml.document 0.0035 0.0036 0.0032
application/x-bibtex-text-file 0.0149 0.0160 0.0155
application/x-bittorrent 0.0002 0.0002 0.0004
application/x-dosexec NaN 0.0000 NaN
application/x-endnote-refer 0.0019 0.0019 0.0020
application/x-mobipocket-ebook 0.0008 0.0013 0.0008
application/x-ms-asx 0.0002 0.0001 0.0002
application/x-msdownload 0.0000 0.0000 0.0000
application/x-pds 0.0005 0.0013 0.0010
application/x-rar-compressed NaN 0.0000 0.0000
application/x-research-info-systems 0.0000 0.0000 0.0000
application/x-sh 0.0010 0.0014 0.0010
application/x-shockwave-flash 0.0000 0.0000 0.0000
application/x-stata-do 0.0004 0.0005 0.0006
application/x-tex 0.0004 0.0003 0.0004
application/x-tex-tfm 0.0003 0.0002 0.0002
application/x-tika-msoffice 0.0025 0.0029 0.0030
application/x-tika-ooxml 0.0024 0.0024 0.0023
application/x-wais-source 0.0001 0.0003 0.0002
application/x-xz NaN 0.0000 NaN
application/xhtml+xml 7.7386 7.4851 7.1630
application/xml 0.0542 0.0552 0.0517
application/zip 0.0000 0.0000 0.0000
application/zlib 0.0001 0.0001 0.0001
audio/mpeg 0.0000 0.0001 0.0000
audio/vnd.wave 0.0000 0.0000 0.0000
audio/x-mpegurl 0.0000 0.0000 0.0000
image/gif 0.0000 0.0000 0.0000
image/jpeg 0.0002 0.0001 0.0002
image/png 0.0000 0.0000 0.0000
image/svg+xml 0.0000 0.0000 0.0000
image/tiff NaN NaN 0.0000
image/vnd.djvu 0.0009 0.0007 0.0008
image/webp 0.0000 0.0000 0.0000
message/rfc822 0.0009 0.0007 0.0008
text/asp 0.0000 0.0000 0.0000
text/aspdotnet NaN 0.0000 0.0000
text/calendar 0.0436 0.0474 0.0463
text/css 0.0007 0.0006 0.0006
text/csv 0.0029 0.0030 0.0028
text/html 90.6458 90.7932 91.2480
text/markdown 0.0093 0.0119 0.0126
text/plain 0.1220 0.1494 0.1270
text/prs.lines.tag 0.0047 0.0053 0.0053
text/tab-separated-values 0.0003 0.0003 0.0003
text/troff 0.0005 0.0006 0.0005
text/turtle 0.0011 0.0012 0.0010
text/vtt 0.0011 0.0012 0.0011
text/x-c++src 0.0002 0.0002 0.0001
text/x-chdr 0.0007 0.0012 0.0012
text/x-csrc 0.0006 0.0006 0.0006
text/x-diff 0.0011 0.0011 0.0010
text/x-jsp 0.0001 0.0001 0.0001
text/x-log 0.0012 0.0012 0.0014
text/x-matlab 0.0024 0.0023 0.0021
text/x-perl 0.0008 0.0008 0.0006
text/x-php 0.0045 0.0030 0.0029
text/x-python 0.0005 0.0005 0.0004
text/x-robots 0.0066 0.0139 0.0030
text/x-vcalendar 0.0004 0.0004 0.0003
text/x-vcard 0.0037 0.0039 0.0039
text/x-web-markdown 0.0141 0.0182 0.0186
text/x-yaml 0.0004 0.0005 0.0007
video/mp4 0.0000 0.0000 0.0000
video/quicktime NaN 0.0000 NaN