MIME Types

The crawled content is dominated by HTML pages and contains only a small percentage of other document formats. The tables show the percentage of the top 100 media or MIME types of the latest monthly crawls.

While the first table is based the Content-Type HTTP header, the second uses the MIME type detected by Apache Tika based on the actual content. The underlying data including page counts is provided in mimetypes.csv resp. mimetypes_detected.csv.

crawl	CC-MAIN-2021-49	CC-MAIN-2022-05	CC-MAIN-2022-21
mimetype
<other>	0.1048	0.0850	0.0620
application/atom+xml	0.1235	0.0818	0.0870
application/download	0.0046	0.0030	0.0028
application/epub+zip	0.0021	0.0020	0.0012
application/force-download	0.0063	0.0045	0.0042
application/javascript	0.0003	0.0002	0.0002
application/json	0.0336	0.0280	0.0231
application/ld+json	0.0011	0.0009	0.0007
application/marc	0.0006	0.0005	0.0004
application/msword	0.0047	0.0034	0.0028
application/octet-stream	0.0733	0.0531	0.0397
application/octetstream	0.0004	0.0003	0.0003
application/pdf	1.3842	0.8378	0.7778
application/pgp-encrypted	0.0002	0.0001	0.0001
application/pgp-signature	0.0030	0.0022	0.0017
application/postscript	0.0004	0.0002	0.0002
application/rdf+xml	0.0051	0.0048	0.0033
application/rss+xml	0.0859	0.0675	0.0590
application/rtf	0.0015	0.0010	0.0011
application/save-to-disk	0.0000	0.0000	0.0000
application/text	0.0028	0.0024	0.0025
application/unknown	0.0005	0.0004	0.0003
application/vnd.android.package-archive	0.0005	0.0004	0.0002
application/vnd.google-earth.kml+xml	0.0022	0.0018	0.0014
application/vnd.google-earth.kmz	0.0004	0.0008	0.0003
application/vnd.ms-excel	0.0032	0.0023	0.0017
application/vnd.ms-powerpoint	0.0014	0.0007	0.0004
application/vnd.ms-word	0.0009	0.0007	0.0005
application/vnd.ms-word.document.macroenabled.12	0.0000	0.0000	0.0000
application/vnd.oasis.opendocument.text	0.0013	0.0009	0.0007
application/vnd.openxmlformats-officedocument.presentationml.presentation	0.0005	0.0003	0.0002
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet	0.0024	0.0018	0.0014
application/vnd.openxmlformats-officedocument.wordprocessingml.document	0.0042	0.0026	0.0022
application/vnd.wap.xhtml+xml	0.0003	0.0002	0.0005
application/x-bibtex	0.0044	0.0035	0.0023
application/x-bittorrent	0.0003	0.0002	0.0002
application/x-debian-package	0.0000	0.0000	0.0000
application/x-download	0.0023	0.0019	0.0012
application/x-endnote-refer	0.0049	0.0042	0.0032
application/x-gzip	0.0018	0.0013	0.0015
application/x-httpd-php	0.0003	0.0003	0.0002
application/x-java-jnlp-file	0.0005	0.0005	0.0003
application/x-javascript	0.0003	0.0002	0.0003
application/x-json	0.0000	0.0000	0.0000
application/x-mobipocket-ebook	0.0002	0.0002	0.0001
application/x-msdownload	0.0004	0.0004	0.0003
application/x-netcdf	0.0010	0.0005	0.0008
application/x-rar-compressed	0.0002	0.0001	0.0001
application/x-research-info-systems	0.0115	0.0095	0.0059
application/x-shockwave-flash	0.0005	0.0003	0.0003
application/x-tar	0.0006	0.0008	0.0005
application/x-tex	0.0003	0.0002	0.0001
application/x-troff-man	0.0006	0.0006	0.0005
application/x-zip-compressed	0.0011	0.0009	0.0006
application/xhtml+xml	0.0654	0.0556	0.0512
application/xml	0.0378	0.0323	0.0273
application/zip	0.0073	0.0066	0.0048
audio/mp3	0.0003	0.0002	0.0002
audio/mp4	0.0000	0.0000	0.0000
audio/mpeg	0.0053	0.0035	0.0031
audio/x-mpegurl	0.0010	0.0009	0.0007
audio/x-pn-realaudio	0.0001	0.0001	0.0001
audio/x-scpls	0.0002	0.0001	0.0001
audio/x-wav	0.0002	0.0002	0.0001
binary/octet-stream	0.0018	0.0007	0.0004
image/gif	0.0082	0.0058	0.0058
image/jp2	0.0000	0.0001	0.0000
image/jpeg	0.2263	0.1490	0.1822
image/jpg	0.0033	0.0025	0.0021
image/pjpeg	0.0009	0.0021	0.0009
image/png	0.0397	0.0253	0.0418
image/svg+xml	0.0012	0.0009	0.0013
image/tiff	0.0009	0.0006	0.0004
image/vnd.djvu	0.0006	0.0005	0.0003
image/webp	0.0007	0.0004	0.0006
message/rfc822	0.0010	0.0005	0.0003
text/calendar	0.0473	0.0343	0.0280
text/css	0.0005	0.0004	0.0004
text/csv	0.0074	0.0057	0.0046
text/enriched	0.0011	0.0010	0.0010
text/html	97.4487	98.2851	98.4112
text/javascript	0.0008	0.0004	0.0004
text/pdf	0.0000	0.0000	0.0000
text/plain	0.1021	0.0778	0.0580
text/tab-separated-values	0.0012	0.0008	0.0007
text/turtle	0.0022	0.0018	0.0012
text/vcard	0.0019	0.0013	0.0012
text/vnd.wap.wml	0.0001	0.0001	0.0001
text/x-bibtex	0.0007	0.0006	0.0007
text/x-c	0.0010	0.0006	0.0002
text/x-csrc	0.0010	0.0009	0.0007
text/x-diff	0.0005	0.0003	0.0003
text/x-patch	0.0009	0.0009	0.0005
text/x-perl	0.0001	0.0001	0.0001
text/x-vcalendar	0.0008	0.0004	0.0004
text/x-vcard	0.0030	0.0021	0.0017
text/xml	0.0845	0.0774	0.0631
unknown/unknown	0.0006	0.0005	0.0002
video/mp4	0.0034	0.0021	0.0026
video/webm	0.0001	0.0001	0.0001
video/x-ms-asf	0.0003	0.0003	0.0002

crawl	CC-MAIN-2021-49	CC-MAIN-2022-05	CC-MAIN-2022-21
mimetype_detected
<other>	0.0222	0.0173	0.0138
application/atom+xml	0.1292	0.0869	0.0920
application/epub+zip	0.0027	0.0025	0.0015
application/gzip	0.0032	0.0028	0.0024
application/javascript	0.0015	0.0008	0.0009
application/json	0.0323	0.0271	0.0223
application/marc	0.0005	0.0017	0.0014
application/mbox	0.0058	0.0043	0.0032
application/msword	0.0037	0.0024	0.0020
application/octet-stream	0.0314	0.0250	0.0241
application/pdf	1.4108	0.8524	0.7846
application/pgp-signature	0.0034	0.0029	0.0020
application/pkcs7-signature	0.0005	0.0003	0.0004
application/postscript	0.0003	0.0001	0.0001
application/rdf+xml	0.0121	0.0115	0.0083
application/rss+xml	0.1471	0.1193	0.1009
application/rtf	0.0027	0.0020	0.0019
application/text	0.0008	0.0004	0.0004
application/vnd.android.package-archive	0.0003	0.0002	0.0002
application/vnd.google-earth.kml+xml	0.0029	0.0025	0.0023
application/vnd.google-earth.kmz	0.0005	0.0008	0.0003
application/vnd.ms-excel	0.0024	0.0016	0.0012
application/vnd.ms-powerpoint	0.0014	0.0007	0.0004
application/vnd.oasis.opendocument.text	0.0013	0.0011	0.0008
application/vnd.openxmlformats-officedocument.presentationml.presentation	0.0004	0.0003	0.0002
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet	0.0023	0.0017	0.0014
application/vnd.openxmlformats-officedocument.wordprocessingml.document	0.0041	0.0026	0.0021
application/x-bibtex-text-file	0.0106	0.0086	0.0064
application/x-bittorrent	0.0004	0.0003	0.0003
application/x-bzip2	0.0013	0.0007	0.0005
application/x-debian-package	0.0001	0.0000	0.0000
application/x-dosexec	0.0003	0.0002	0.0002
application/x-endnote-refer	0.0031	0.0025	0.0017
application/x-hdf	0.0026	0.0005	0.0007
application/x-mobipocket-ebook	0.0005	0.0005	0.0003
application/x-ms-asx	0.0003	0.0002	0.0002
application/x-msdownload	0.0002	0.0001	0.0002
application/x-rar-compressed	0.0000	0.0000	0.0000
application/x-research-info-systems	0.0003	0.0002	0.0002
application/x-sh	0.0016	0.0015	0.0013
application/x-shockwave-flash	0.0005	0.0003	0.0003
application/x-stata-do	0.0012	0.0013	0.0014
application/x-tika-msoffice	0.0046	0.0033	0.0026
application/x-tika-ooxml	0.0043	0.0029	0.0026
application/x-wais-source	0.0004	0.0003	0.0002
application/x-xz	0.0006	0.0007	0.0002
application/xhtml+xml	14.6296	14.0628	12.5547
application/xml	0.0817	0.0778	0.0658
application/zip	0.0132	0.0108	0.0081
application/zlib	0.0002	0.0002	0.0004
application/zstd	0.0014	0.0018	0.0008
audio/midi	0.0001	0.0000	0.0001
audio/mp4	0.0001	0.0001	0.0000
audio/mpeg	0.0080	0.0054	0.0046
audio/vnd.wave	0.0008	0.0007	0.0004
audio/x-mpegurl	0.0003	0.0003	0.0000
image/bmp	0.0001	0.0001	0.0001
image/gif	0.0065	0.0049	0.0050
image/jpeg	0.2421	0.1626	0.1906
image/png	0.0393	0.0249	0.0416
image/svg+xml	0.0013	0.0010	0.0015
image/tiff	0.0011	0.0018	0.0005
image/vnd.djvu	0.0010	0.0007	0.0004
image/vnd.dxf; format=ascii	0.0006	0.0004	0.0003
image/vnd.microsoft.icon	0.0004	0.0002	0.0001
image/webp	0.0008	0.0004	0.0007
message/rfc822	0.0039	0.0031	0.0028
text/calendar	0.0511	0.0366	0.0306
text/css	0.0005	0.0004	0.0004
text/csv	0.0065	0.0050	0.0044
text/html	82.8554	84.2471	85.8782
text/plain	0.1446	0.1192	0.0870
text/prs.lines.tag	0.0053	0.0035	0.0026
text/tab-separated-values	0.0012	0.0009	0.0008
text/troff	0.0007	0.0007	0.0006
text/turtle	0.0020	0.0016	0.0012
text/vtt	0.0012	0.0007	0.0005
text/x-c++src	0.0009	0.0007	0.0005
text/x-chdr	0.0020	0.0016	0.0011
text/x-csrc	0.0031	0.0026	0.0019
text/x-diff	0.0029	0.0023	0.0016
text/x-jsp	0.0002	0.0001	0.0001
text/x-log	0.0050	0.0028	0.0018
text/x-matlab	0.0037	0.0033	0.0024
text/x-patch	0.0003	0.0003	0.0002
text/x-perl	0.0039	0.0027	0.0026
text/x-php	0.0054	0.0036	0.0040
text/x-python	0.0010	0.0010	0.0008
text/x-vcalendar	0.0009	0.0005	0.0005
text/x-vcard	0.0060	0.0042	0.0037
text/x-web-markdown	0.0006	0.0006	0.0005
video/mp4	0.0036	0.0022	0.0026
video/quicktime	0.0004	0.0003	0.0004
video/webm	0.0001	0.0001	0.0001
video/x-m4v	0.0001	0.0001	0.0001
video/x-matroska	0.0000	0.0000	0.0000
video/x-ms-wmv	0.0001	0.0001	0.0001