MIME Types

The crawled content is dominated by HTML pages and contains only a small percentage of other document formats. The tables show the percentage of the top 100 media or MIME types of the latest monthly crawls.

While the first table is based the Content-Type HTTP header, the second uses the MIME type detected by Apache Tika based on the actual content. The underlying data including page counts is provided in mimetypes.csv resp. mimetypes_detected.csv.

crawl	CC-MAIN-2022-27	CC-MAIN-2022-33	CC-MAIN-2022-40
mimetype
<other>	0.0730	0.1141	0.0687
application/atom+xml	0.1081	0.1087	0.0703
application/download	0.0037	0.0040	0.0021
application/epub+zip	0.0015	0.0022	0.0012
application/force-download	0.0057	0.0079	0.0046
application/javascript	0.0002	0.0003	0.0002
application/json	0.0254	0.0314	0.0243
application/ld+json	0.0010	0.0012	0.0013
application/marc	0.0004	0.0004	0.0004
application/msword	0.0036	0.0044	0.0028
application/octet-stream	0.0538	0.0647	0.0482
application/octetstream	0.0003	0.0005	0.0003
application/pdf	0.9942	1.1008	0.7067
application/pgp-encrypted	0.0001	0.0002	0.0001
application/pgp-signature	0.0020	0.0024	0.0021
application/postscript	0.0002	0.0003	0.0001
application/rdf+xml	0.0040	0.0051	0.0042
application/rss+xml	0.0694	0.0769	0.0562
application/rtf	0.0011	0.0016	0.0011
application/save-to-disk	0.0000	0.0000	0.0000
application/text	0.0024	0.0029	0.0023
application/unknown	0.0003	0.0005	0.0002
application/vnd.android.package-archive	0.0003	0.0002	0.0001
application/vnd.google-earth.kml+xml	0.0015	0.0017	0.0015
application/vnd.google-earth.kmz	0.0003	0.0004	0.0003
application/vnd.ms-excel	0.0025	0.0029	0.0018
application/vnd.ms-powerpoint	0.0005	0.0007	0.0003
application/vnd.ms-word	0.0006	0.0008	0.0005
application/vnd.ms-word.document.macroenabled.12	0.0000	0.0000	0.0000
application/vnd.oasis.opendocument.text	0.0009	0.0011	0.0007
application/vnd.openxmlformats-officedocument.presentationml.presentation	0.0003	0.0004	0.0002
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet	0.0018	0.0023	0.0017
application/vnd.openxmlformats-officedocument.wordprocessingml.document	0.0031	0.0040	0.0024
application/vnd.wap.xhtml+xml	0.0007	0.0010	0.0009
application/x-bibtex	0.0029	0.0039	0.0031
application/x-bittorrent	0.0002	0.0003	0.0002
application/x-debian-package	0.0000	0.0000	0.0000
application/x-download	0.0016	0.0021	0.0017
application/x-endnote-refer	0.0033	0.0040	0.0034
application/x-gzip	0.0014	0.0019	0.0013
application/x-httpd-php	0.0002	0.0005	0.0002
application/x-java-jnlp-file	0.0001	0.0002	0.0003
application/x-javascript	0.0003	0.0004	0.0004
application/x-json	0.0000	0.0000	0.0000
application/x-mobipocket-ebook	0.0002	0.0002	0.0002
application/x-msdownload	0.0004	0.0006	0.0005
application/x-netcdf	0.0017	0.0022	0.0015
application/x-rar-compressed	0.0001	0.0002	0.0002
application/x-research-info-systems	0.0071	0.0084	0.0069
application/x-shockwave-flash	0.0003	0.0003	0.0002
application/x-tar	0.0004	0.0004	0.0007
application/x-tex	0.0001	0.0002	0.0002
application/x-troff-man	0.0008	0.0008	0.0008
application/x-zip-compressed	0.0008	0.0010	0.0005
application/xhtml+xml	0.0496	0.0632	0.0486
application/xml	0.0320	0.0353	0.0264
application/zip	0.0063	0.0084	0.0061
audio/mp3	0.0003	0.0004	0.0002
audio/mp4	0.0000	0.0000	0.0000
audio/mpeg	0.0041	0.0046	0.0031
audio/x-mpegurl	0.0010	0.0014	0.0009
audio/x-pn-realaudio	0.0001	0.0001	0.0000
audio/x-scpls	0.0002	0.0001	0.0001
audio/x-wav	0.0002	0.0003	0.0002
binary/octet-stream	0.0014	0.0006	0.0005
image/gif	0.0047	0.0062	0.0042
image/jp2	0.0000	0.0000	0.0000
image/jpeg	0.1772	0.2498	0.1559
image/jpg	0.0028	0.0031	0.0024
image/pjpeg	0.0011	0.0009	0.0005
image/png	0.0354	0.0482	0.0313
image/svg+xml	0.0012	0.0017	0.0010
image/tiff	0.0007	0.0007	0.0005
image/vnd.djvu	0.0003	0.0006	0.0004
image/webp	0.0007	0.0010	0.0010
message/rfc822	0.0003	0.0003	0.0003
text/calendar	0.0358	0.0486	0.0341
text/css	0.0005	0.0004	0.0005
text/csv	0.0046	0.0059	0.0041
text/enriched	0.0008	0.0011	0.0008
text/html	98.1070	97.7756	98.5070
text/javascript	0.0003	0.0006	0.0002
text/pdf	0.0000	0.0000	0.0000
text/plain	0.0648	0.0785	0.0649
text/tab-separated-values	0.0008	0.0010	0.0006
text/turtle	0.0013	0.0016	0.0015
text/vcard	0.0012	0.0014	0.0009
text/vnd.wap.wml	0.0001	0.0001	0.0001
text/x-bibtex	0.0006	0.0007	0.0005
text/x-c	0.0003	0.0003	0.0002
text/x-csrc	0.0006	0.0009	0.0007
text/x-diff	0.0002	0.0003	0.0002
text/x-patch	0.0007	0.0008	0.0004
text/x-perl	0.0001	0.0001	0.0001
text/x-vcalendar	0.0005	0.0006	0.0004
text/x-vcard	0.0022	0.0026	0.0018
text/xml	0.0700	0.0757	0.0651
unknown/unknown	0.0003	0.0003	0.0002
video/mp4	0.0022	0.0025	0.0020
video/webm	0.0002	0.0002	0.0001
video/x-ms-asf	0.0002	0.0002	0.0001

crawl	CC-MAIN-2022-27	CC-MAIN-2022-33	CC-MAIN-2022-40
mimetype_detected
<other>	0.0160	0.0195	0.0151
application/atom+xml	0.1136	0.1144	0.0754
application/epub+zip	0.0019	0.0027	0.0015
application/gpx+xml	0.0005	0.0005	0.0003
application/gzip	0.0025	0.0042	0.0032
application/javascript	0.0008	0.0014	0.0007
application/json	0.0246	0.0305	0.0236
application/marc	0.0015	0.0017	0.0014
application/mbox	0.0039	0.0052	0.0036
application/msword	0.0025	0.0031	0.0019
application/octet-stream	0.0207	0.0274	0.0266
application/pdf	1.0113	1.1169	0.7135
application/pgp-signature	0.0029	0.0028	0.0028
application/pkcs7-signature	0.0005	0.0005	0.0003
application/postscript	0.0001	0.0002	0.0001
application/rdf+xml	0.0099	0.0117	0.0091
application/rss+xml	0.1156	0.1287	0.0973
application/rtf	0.0018	0.0025	0.0019
application/text	0.0004	0.0005	0.0005
application/vnd.android.package-archive	0.0003	0.0003	0.0002
application/vnd.google-earth.kml+xml	0.0024	0.0024	0.0025
application/vnd.google-earth.kmz	0.0003	0.0004	0.0003
application/vnd.ms-excel	0.0018	0.0020	0.0013
application/vnd.ms-powerpoint	0.0005	0.0006	0.0002
application/vnd.oasis.opendocument.text	0.0013	0.0013	0.0008
application/vnd.openxmlformats-officedocument.presentationml.presentation	0.0004	0.0004	0.0002
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet	0.0018	0.0023	0.0016
application/vnd.openxmlformats-officedocument.wordprocessingml.document	0.0030	0.0038	0.0023
application/x-bibtex-text-file	0.0063	0.0087	0.0069
application/x-bittorrent	0.0003	0.0004	0.0003
application/x-bzip2	0.0014	0.0003	0.0015
application/x-debian-package	0.0000	0.0001	0.0001
application/x-dosexec	0.0003	0.0004	0.0002
application/x-endnote-refer	0.0020	0.0024	0.0019
application/x-hdf	0.0015	0.0020	0.0014
application/x-mobipocket-ebook	0.0004	0.0005	0.0003
application/x-ms-asx	0.0002	0.0002	0.0001
application/x-msdownload	0.0002	0.0002	0.0002
application/x-rar-compressed	0.0000	0.0000	0.0000
application/x-research-info-systems	0.0002	0.0001	0.0002
application/x-sh	0.0013	0.0015	0.0012
application/x-shockwave-flash	0.0003	0.0003	0.0002
application/x-stata-do	0.0012	0.0013	0.0018
application/x-tex	0.0003	0.0004	0.0004
application/x-tika-msoffice	0.0033	0.0044	0.0027
application/x-tika-ooxml	0.0035	0.0038	0.0027
application/x-wais-source	0.0003	0.0003	0.0003
application/x-xz	0.0004	0.0012	0.0003
application/xhtml+xml	12.7558	13.2699	12.0352
application/xml	0.0721	0.0777	0.0676
application/zip	0.0105	0.0136	0.0099
application/zlib	0.0003	0.0003	0.0002
application/zstd	0.0010	0.0014	0.0015
audio/midi	0.0001	0.0001	0.0001
audio/mp4	0.0000	0.0001	0.0001
audio/mpeg	0.0062	0.0075	0.0049
audio/vnd.wave	0.0006	0.0007	0.0006
audio/x-mpegurl	0.0001	0.0002	0.0002
image/bmp	0.0001	0.0001	0.0001
image/gif	0.0040	0.0056	0.0037
image/jpeg	0.1883	0.2629	0.1645
image/png	0.0349	0.0478	0.0309
image/svg+xml	0.0015	0.0020	0.0013
image/tiff	0.0011	0.0017	0.0006
image/vnd.djvu	0.0004	0.0008	0.0005
image/vnd.dxf; format=ascii	0.0005	0.0005	0.0003
image/vnd.microsoft.icon	0.0001	0.0001	0.0001
image/webp	0.0008	0.0010	0.0012
message/rfc822	0.0028	0.0042	0.0028
text/asp	0.0000	NaN	NaN
text/calendar	0.0397	0.0532	0.0371
text/css	0.0005	0.0004	0.0005
text/csv	0.0050	0.0058	0.0040
text/html	85.3677	84.5605	86.4851
text/plain	0.1065	0.1245	0.1052
text/prs.lines.tag	0.0035	0.0047	0.0041
text/tab-separated-values	0.0009	0.0010	0.0007
text/troff	0.0006	0.0008	0.0006
text/turtle	0.0012	0.0015	0.0014
text/vtt	0.0007	0.0008	0.0006
text/x-c++src	0.0004	0.0005	0.0003
text/x-chdr	0.0011	0.0012	0.0009
text/x-csrc	0.0017	0.0021	0.0017
text/x-diff	0.0015	0.0021	0.0018
text/x-jsp	0.0001	0.0001	0.0001
text/x-log	0.0024	0.0039	0.0021
text/x-matlab	0.0033	0.0032	0.0024
text/x-perl	0.0024	0.0034	0.0030
text/x-php	0.0045	0.0049	0.0037
text/x-python	0.0007	0.0010	0.0007
text/x-vcalendar	0.0005	0.0006	0.0005
text/x-vcard	0.0043	0.0050	0.0033
text/x-web-markdown	0.0004	0.0006	0.0004
video/mp4	0.0024	0.0027	0.0021
video/quicktime	0.0003	0.0005	0.0004
video/webm	0.0002	0.0002	0.0001
video/x-m4v	0.0001	0.0001	0.0000
video/x-matroska	0.0000	0.0000	0.0000