Statistics of Common Crawl Monthly Archives

Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives
Latest crawl: CC-MAIN-2026-43

View the Project on GitHub

MIME Types

The crawled content is dominated by HTML pages and contains only a small percentage of other document formats. The tables show the percentage of the top 100 media or MIME types of the latest monthly crawls.

While the first table is based the Content-Type HTTP header, the second uses the MIME type detected by Apache Tika based on the actual content. The underlying data including page counts is provided in mimetypes.csv resp. mimetypes_detected.csv.

crawl CC-MAIN-2026-34 CC-MAIN-2026-39 CC-MAIN-2026-43
mimetype
<other> 0.0646 0.0706 0.0679
application/atom+xml 0.1804 0.1570 0.1494
application/calendar 0.0002 0.0002 0.0001
application/download 0.0047 0.0043 0.0044
application/epub+zip 0.0026 0.0019 0.0024
application/force-download 0.0124 0.0118 0.0133
application/gpx+xml 0.0011 0.0011 0.0012
application/ics 0.0005 0.0004 0.0005
application/javascript 0.0017 0.0012 0.0006
application/json 0.0272 0.0298 0.0280
application/ld+json 0.0013 0.0012 0.0014
application/marc 0.0003 0.0003 0.0004
application/msword 0.0029 0.0028 0.0028
application/octet-stream 0.0498 0.0493 0.0504
application/octetstream 0.0003 0.0003 0.0001
application/pdf 1.0160 0.9497 1.0025
application/pgp-encrypted 0.0002 0.0001 0.0001
application/pgp-signature 0.0023 0.0028 0.0029
application/postscript 0.0002 0.0002 0.0002
application/rdf+xml 0.0029 0.0029 0.0029
application/rss+xml 0.0453 0.0435 0.0447
application/rtf 0.0008 0.0007 0.0005
application/save-to-disk 0.0000 0.0000 0.0000
application/text 0.0002 0.0001 0.0001
application/unknown 0.0003 0.0002 0.0002
application/vnd.android.package-archive 0.0000 0.0000 0.0000
application/vnd.google-earth.kml+xml 0.0013 0.0013 0.0014
application/vnd.google-earth.kmz 0.0004 0.0003 0.0003
application/vnd.ms-excel 0.0013 0.0013 0.0015
application/vnd.ms-powerpoint 0.0001 0.0001 0.0001
application/vnd.ms-word 0.0002 0.0002 0.0002
application/vnd.oasis.opendocument.text 0.0011 0.0009 0.0009
application/vnd.openxmlformats-officedocument.presentationml.presentation 0.0003 0.0003 0.0002
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet 0.0018 0.0017 0.0019
application/vnd.openxmlformats-officedocument.wordprocessingml.document 0.0036 0.0032 0.0032
application/vnd.wap.xhtml+xml 0.0006 0.0007 0.0002
application/x-bibtex 0.0105 0.0107 0.0114
application/x-bittorrent 0.0001 0.0003 0.0001
application/x-debian-package 0.0000 0.0000 0.0000
application/x-download 0.0027 0.0033 0.0035
application/x-endnote-refer 0.0005 0.0005 0.0009
application/x-gzip 0.0002 0.0002 0.0003
application/x-httpd-php 0.0003 0.0003 0.0005
application/x-java-jnlp-file 0.0001 0.0000 0.0000
application/x-javascript 0.0003 0.0002 0.0001
application/x-json 0.0000 0.0000 0.0000
application/x-mobipocket-ebook 0.0010 0.0006 0.0005
application/x-msdownload 0.0003 0.0003 0.0003
application/x-netcdf 0.0007 0.0001 0.0001
application/x-research-info-systems 0.0127 0.0128 0.0139
application/x-shockwave-flash 0.0000 0.0000 0.0000
application/x-tar 0.0001 0.0001 0.0001
application/x-tex 0.0001 0.0002 0.0001
application/x-troff-man 0.0004 0.0004 0.0004
application/x-zip-compressed 0.0000 0.0000 0.0000
application/xhtml+xml 0.0088 0.0082 0.0073
application/xml 0.0253 0.0247 0.0262
application/zip 0.0002 0.0002 0.0001
audio/mpeg 0.0002 0.0000 0.0000
audio/x-mpegurl 0.0005 0.0005 0.0005
audio/x-scpls 0.0001 0.0001 0.0001
audio/x-wav 0.0000 0.0000 0.0000
binary/octet-stream 0.0005 0.0005 0.0005
image/gif 0.0002 0.0001 0.0002
image/jp2 0.0000 0.0000 0.0000
image/jpeg 0.0002 0.0003 0.0004
image/jpg 0.0000 0.0000 0.0000
image/pjpeg 0.0000 0.0000 0.0000
image/png 0.0001 0.0000 0.0000
image/svg+xml 0.0000 0.0000 0.0000
image/tiff 0.0000 0.0000 0.0000
image/vnd.djvu 0.0005 0.0004 0.0005
image/webp 0.0000 0.0000 0.0000
message/rfc822 0.0002 0.0003 0.0001
text/calendar 0.0383 0.0376 0.0401
text/css 0.0006 0.0006 0.0005
text/csv 0.0031 0.0028 0.0030
text/directory 0.0002 0.0002 0.0002
text/enriched 0.0001 0.0000 0.0000
text/html 98.2554 98.3804 98.3103
text/javascript 0.0003 0.0003 0.0003
text/markdown 0.0289 0.0288 0.0715
text/pdf 0.0000 0.0000 0.0000
text/plain 0.1181 0.0861 0.0618
text/prs.lines.tag 0.0033 0.0037 0.0036
text/tab-separated-values 0.0003 0.0003 0.0003
text/turtle 0.0012 0.0010 0.0010
text/vcard 0.0013 0.0014 0.0016
text/x-bibtex 0.0006 0.0005 0.0006
text/x-c 0.0001 0.0001 0.0001
text/x-csrc 0.0002 0.0001 0.0002
text/x-diff 0.0003 0.0003 0.0003
text/x-patch 0.0002 0.0002 0.0002
text/x-perl 0.0001 0.0000 0.0000
text/x-vcalendar 0.0004 0.0004 0.0004
text/x-vcard 0.0020 0.0020 0.0020
text/xml 0.0493 0.0449 0.0462
unknown/unknown 0.0000 0.0000 0.0000
video/mp4 0.0000 0.0000 0.0000
video/x-ms-asf 0.0001 0.0002 0.0002

crawl CC-MAIN-2026-34 CC-MAIN-2026-39 CC-MAIN-2026-43
mimetype_detected
<other> 0.0157 0.0145 0.0146
application/atom+xml 0.1817 0.1583 0.1508
application/epub+zip 0.0030 0.0023 0.0027
application/gpx+xml 0.0011 0.0011 0.0012
application/gzip 0.0000 0.0000 0.0000
application/json 0.0266 0.0260 0.0267
application/marc 0.0008 0.0006 0.0007
application/mbox 0.0012 0.0011 0.0011
application/msword 0.0021 0.0021 0.0021
application/octet-stream 0.0116 0.0154 0.0131
application/pdf 1.0307 0.9647 1.0188
application/pgp-signature 0.0024 0.0025 0.0031
application/pkcs7-signature 0.0008 0.0007 0.0006
application/postscript 0.0001 0.0001 0.0001
application/rdf+xml 0.0063 0.0060 0.0063
application/rss+xml 0.0702 0.0661 0.0677
application/rtf 0.0014 0.0012 0.0009
application/text 0.0000 0.0000 0.0000
application/vnd.android.package-archive 0.0000 0.0000 0.0000
application/vnd.google-earth.kml+xml 0.0025 0.0027 0.0029
application/vnd.google-earth.kmz 0.0004 0.0004 0.0004
application/vnd.ms-excel 0.0009 0.0008 0.0009
application/vnd.ms-powerpoint 0.0001 0.0001 0.0001
application/vnd.oasis.opendocument.spreadsheet 0.0006 0.0005 0.0005
application/vnd.oasis.opendocument.text 0.0013 0.0010 0.0011
application/vnd.openxmlformats-officedocument.presentationml.presentation 0.0003 0.0003 0.0003
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet 0.0018 0.0017 0.0019
application/vnd.openxmlformats-officedocument.wordprocessingml.document 0.0036 0.0032 0.0033
application/x-bibtex-text-file 0.0160 0.0155 0.0166
application/x-bittorrent 0.0002 0.0004 0.0002
application/x-dosexec 0.0000 NaN NaN
application/x-endnote-refer 0.0019 0.0020 0.0025
application/x-mobipocket-ebook 0.0013 0.0008 0.0006
application/x-ms-asx 0.0001 0.0002 0.0002
application/x-msdownload 0.0000 0.0000 0.0000
application/x-pds 0.0013 0.0010 0.0013
application/x-rar-compressed 0.0000 0.0000 0.0000
application/x-research-info-systems 0.0000 0.0000 0.0000
application/x-sh 0.0014 0.0010 0.0012
application/x-shockwave-flash 0.0000 0.0000 0.0000
application/x-stata-do 0.0005 0.0006 0.0005
application/x-tex 0.0003 0.0004 0.0003
application/x-tex-tfm 0.0002 0.0002 0.0005
application/x-tika-msoffice 0.0029 0.0030 0.0031
application/x-tika-ooxml 0.0024 0.0023 0.0022
application/x-wais-source 0.0003 0.0002 0.0001
application/x-xz 0.0000 NaN NaN
application/xhtml+xml 7.4851 7.1630 6.9394
application/xml 0.0552 0.0517 0.0535
application/zip 0.0000 0.0000 0.0000
application/zlib 0.0001 0.0001 0.0001
audio/mpeg 0.0001 0.0000 0.0000
audio/vnd.wave 0.0000 0.0000 0.0000
audio/x-mpegurl 0.0000 0.0000 0.0000
image/gif 0.0000 0.0000 0.0000
image/jpeg 0.0001 0.0002 0.0002
image/png 0.0000 0.0000 0.0000
image/svg+xml 0.0000 0.0000 0.0000
image/tiff NaN 0.0000 NaN
image/vnd.djvu 0.0007 0.0008 0.0008
image/webp 0.0000 0.0000 0.0000
message/rfc822 0.0007 0.0008 0.0006
text/asp 0.0000 0.0000 0.0000
text/aspdotnet 0.0000 0.0000 0.0000
text/calendar 0.0474 0.0463 0.0494
text/css 0.0006 0.0006 0.0005
text/csv 0.0030 0.0028 0.0030
text/html 90.7932 91.2480 91.3972
text/markdown 0.0119 0.0126 0.0283
text/plain 0.1494 0.1270 0.1061
text/prs.lines.tag 0.0053 0.0053 0.0067
text/tab-separated-values 0.0003 0.0003 0.0003
text/troff 0.0006 0.0005 0.0005
text/turtle 0.0012 0.0010 0.0010
text/vtt 0.0012 0.0011 0.0012
text/x-c++src 0.0002 0.0001 0.0001
text/x-chdr 0.0012 0.0012 0.0009
text/x-csrc 0.0006 0.0006 0.0006
text/x-diff 0.0011 0.0010 0.0010
text/x-jsp 0.0001 0.0001 0.0001
text/x-log 0.0012 0.0014 0.0015
text/x-matlab 0.0023 0.0021 0.0022
text/x-perl 0.0008 0.0006 0.0011
text/x-php 0.0030 0.0029 0.0025
text/x-python 0.0005 0.0004 0.0005
text/x-robots 0.0139 0.0030 0.0003
text/x-vcalendar 0.0004 0.0003 0.0003
text/x-vcard 0.0039 0.0039 0.0041
text/x-web-markdown 0.0182 0.0186 0.0439
text/x-yaml 0.0005 0.0007 0.0006
video/mp4 0.0000 0.0000 0.0000
video/quicktime 0.0000 NaN 0.0000