MIME Types

The crawled content is dominated by HTML pages and contains only a small percentage of other document formats. The tables show the percentage of the top 100 media or MIME types of the latest monthly crawls.

While the first table is based the Content-Type HTTP header, the second uses the MIME type detected by Apache Tika based on the actual content.

crawl	CC-MAIN-2019-13	CC-MAIN-2019-18	CC-MAIN-2019-22
mimetype
<other>	1.2473	1.3701	1.4161
application/atom+xml	0.1045	0.1050	0.1564
application/download	0.0013	0.0011	0.0015
application/epub+zip	0.0013	0.0012	0.0016
application/force-download	0.0031	0.0027	0.0032
application/javascript	0.0006	0.0005	0.0004
application/json	0.0116	0.0147	0.0165
application/jsonrequest	0.0000	0.0000	nan
application/ld+json	0.0003	0.0003	0.0003
application/marc	0.0001	0.0002	0.0002
application/msword	0.0079	0.0031	0.0040
application/octet-stream	0.0390	0.0382	0.0429
application/octetstream	0.0002	0.0002	0.0003
application/pdf	0.4675	0.5085	0.7069
application/pgp-encrypted	0.0000	0.0000	0.0001
application/pgp-signature	0.0008	0.0016	0.0008
application/postscript	0.0012	0.0012	0.0007
application/rdf+xml	0.0017	0.0027	0.0021
application/rss+xml	0.0739	0.0861	0.0844
application/rtf	0.0016	0.0016	0.0007
application/save-to-disk	0.0000	nan	0.0000
application/text	0.0016	0.0023	0.0018
application/unknown	0.0003	0.0002	0.0003
application/vnd.android.package-archive	0.0009	0.0008	0.0012
application/vnd.google-earth.kml+xml	0.0022	0.0018	0.0018
application/vnd.google-earth.kmz	0.0006	0.0004	0.0004
application/vnd.ms-excel	0.0024	0.0012	0.0016
application/vnd.ms-powerpoint	0.0010	0.0006	0.0009
application/vnd.ms-word	0.0004	0.0005	0.0006
application/vnd.ms-word.document.macroenabled.12	0.0010	0.0005	0.0002
application/vnd.openxmlformats-officedocument.presentationml.presentation	0.0014	0.0015	0.0004
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet	0.0031	0.0030	0.0013
application/vnd.openxmlformats-officedocument.wordprocessingml.document	0.0089	0.0095	0.0029
application/vnd.wap.xhtml+xml	0.0001	0.0001	0.0001
application/x-bibtex	0.0013	0.0020	0.0017
application/x-bittorrent	0.0007	0.0006	0.0004
application/x-debian-package	0.0002	0.0003	0.0001
application/x-download	0.0005	0.0004	0.0008
application/x-dvi	0.0001	0.0000	0.0000
application/x-endnote-refer	0.0020	0.0030	0.0025
application/x-gzip	0.0048	0.0008	0.0006
application/x-httpd-php	0.0007	0.0007	0.0006
application/x-java-jnlp-file	0.0001	0.0001	0.0002
application/x-javascript	0.0003	0.0002	0.0003
application/x-json	0.0000	0.0000	0.0000
application/x-mobipocket-ebook	0.0002	0.0002	0.0002
application/x-msdownload	0.0005	0.0003	0.0004
application/x-netcdf	0.0002	0.0004	0.0002
application/x-rar-compressed	0.0014	0.0021	0.0004
application/x-research-info-systems	0.0028	0.0046	0.0045
application/x-shockwave-flash	0.0061	0.0012	0.0009
application/x-tar	0.0011	0.0006	0.0003
application/x-tex	0.0002	0.0001	0.0001
application/x-troff-man	0.0003	0.0003	0.0002
application/x-xz	0.0008	0.0009	0.0010
application/x-zip-compressed	0.0017	0.0005	0.0005
application/xhtml+xml	0.0185	0.0248	0.0246
application/xml	0.0281	0.0354	0.0292
application/zip	0.0089	0.0031	0.0035
audio/mp3	0.0005	0.0002	0.0003
audio/mpeg	0.0207	0.0034	0.0048
audio/x-mpegurl	0.0020	0.0016	0.0014
audio/x-pn-realaudio	0.0008	0.0013	0.0001
audio/x-scpls	0.0002	0.0002	0.0002
audio/x-wav	0.0012	0.0014	0.0005
binary/octet-stream	0.0007	0.0006	0.0007
image/gif	0.0131	0.0026	0.0032
image/jp2	0.0001	0.0001	0.0002
image/jpeg	0.5469	0.1323	0.1458
image/jpg	0.0039	0.0015	0.0026
image/pjpeg	0.0003	0.0002	0.0004
image/png	0.0629	0.0240	0.0246
image/svg+xml	0.0020	0.0005	0.0004
image/tiff	0.0010	0.0003	0.0002
image/vnd.djvu	0.0008	0.0005	0.0005
image/webp	0.0006	0.0006	0.0008
message/rfc822	0.0002	0.0001	0.0002
text/calendar	0.0321	0.0324	0.0356
text/csv	0.0018	0.0019	0.0015
text/enriched	0.0005	0.0009	0.0007
text/html	97.0932	97.4066	97.1164
text/javascript	0.0004	0.0003	0.0005
text/pdf	0.0000	0.0000	0.0000
text/plain	0.0481	0.0464	0.0417
text/tab-separated-values	0.0004	0.0004	0.0006
text/turtle	0.0004	0.0004	0.0004
text/vnd.wap.wml	0.0002	0.0002	0.0002
text/x-bibtex	0.0003	0.0003	0.0002
text/x-c	0.0002	0.0002	0.0001
text/x-csrc	0.0001	0.0004	0.0004
text/x-diff	0.0003	0.0003	0.0004
text/x-patch	0.0001	0.0002	0.0002
text/x-perl	0.0001	0.0001	0.0000
text/x-vcalendar	0.0009	0.0007	0.0008
text/x-vcard	0.0018	0.0014	0.0016
text/xml	0.0825	0.0892	0.0835
unknown/unknown	0.0004	0.0004	0.0004
video/mp4	0.0067	0.0010	0.0013
video/webm	0.0018	0.0001	0.0001
video/x-ms-asf	0.0007	0.0005	0.0004
video/x-ms-wvx	0.0001	0.0000	0.0000

crawl	CC-MAIN-2019-13	CC-MAIN-2019-18	CC-MAIN-2019-22
mimetype_detected
<other>	0.0206	0.0202	0.0155
application/atom+xml	0.1102	0.1104	0.1614
application/download	0.0001	0.0001	0.0001
application/epub+zip	0.0018	0.0016	0.0020
application/force-download	0.0001	0.0005	0.0005
application/gzip	0.0108	0.0025	0.0023
application/javascript	0.0012	0.0011	0.0011
application/json	0.0116	0.0148	0.0167
application/ld+json	nan	0.0000	0.0000
application/marc	0.0000	0.0000	0.0001
application/mbox	0.0007	0.0009	0.0013
application/msword	0.0071	0.0023	0.0034
application/n-triples	0.0000	nan	nan
application/octet-stream	0.0067	0.0102	0.0133
application/pdf	0.4782	0.5189	0.7248
application/pgp-signature	0.0014	0.0018	0.0009
application/pls+xml	0.0003	0.0003	0.0002
application/postscript	0.0012	0.0011	0.0006
application/rdf+xml	0.0076	0.0108	0.0097
application/rss+xml	0.1386	0.1527	0.1472
application/rtf	0.0029	0.0031	0.0020
application/text	0.0002	0.0002	0.0002
application/vnd.android.package-archive	0.0012	0.0012	0.0014
application/vnd.google-earth.kml+xml	0.0026	0.0023	0.0022
application/vnd.google-earth.kmz	0.0006	0.0004	0.0004
application/vnd.ms-excel	0.0022	0.0010	0.0014
application/vnd.ms-powerpoint	0.0010	0.0006	0.0008
application/vnd.oasis.opendocument.text	0.0008	0.0006	0.0007
application/vnd.openxmlformats-officedocument.presentationml.presentation	0.0015	0.0017	0.0005
application/vnd.openxmlformats-officedocument.presentationml.slideshow	0.0003	0.0002	0.0002
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet	0.0032	0.0032	0.0013
application/vnd.openxmlformats-officedocument.wordprocessingml.document	0.0097	0.0105	0.0031
application/x-bibtex-text-file	0.0028	0.0031	0.0031
application/x-bittorrent	0.0008	0.0007	0.0005
application/x-bzip2	0.0020	0.0010	0.0004
application/x-debian-package	0.0012	0.0009	0.0007
application/x-dosexec	0.0011	0.0004	0.0004
application/x-endnote-refer	0.0009	0.0012	0.0012
application/x-go-sgf	0.0000	0.0000	0.0000
application/x-mobipocket-ebook	0.0006	0.0005	0.0005
application/x-ms-asx	0.0006	0.0005	0.0004
application/x-msdownload	0.0003	0.0003	0.0002
application/x-rar-compressed	0.0029	0.0036	0.0010
application/x-research-info-systems	0.0000	0.0001	0.0001
application/x-sh	0.0006	0.0006	0.0008
application/x-shockwave-flash	0.0062	0.0013	0.0009
application/x-tika-msoffice	0.0030	0.0028	0.0024
application/x-tika-ooxml	0.0022	0.0019	0.0018
application/x-wais-source	0.0004	0.0003	0.0004
application/x-xz	0.0024	0.0029	0.0021
application/xhtml+xml	23.3495	22.5811	22.5918
application/xml	0.0567	0.0668	0.0607
application/zip	0.0121	0.0052	0.0068
application/zlib	0.0009	0.0007	0.0012
audio/midi	0.0014	0.0021	0.0003
audio/mp4	0.0011	0.0019	0.0016
audio/mpeg	0.0224	0.0041	0.0062
audio/vnd.wave	0.0018	0.0021	0.0009
audio/vorbis	0.0016	0.0013	0.0002
audio/x-mpegurl	0.0003	0.0002	0.0002
audio/x-ms-wma	0.0002	0.0000	0.0000
image/bmp	0.0006	0.0008	0.0002
image/gif	0.0130	0.0027	0.0034
image/jp2	0.0001	0.0001	0.0002
image/jpeg	0.5565	0.1390	0.1552
image/png	0.0632	0.0235	0.0236
image/svg+xml	0.0021	0.0005	0.0005
image/tiff	0.0011	0.0003	0.0004
image/vnd.djvu	0.0012	0.0007	0.0007
image/vnd.microsoft.icon	0.0010	0.0001	0.0001
image/webp	0.0007	0.0007	0.0008
message/rfc822	0.0016	0.0014	0.0012
text/asp	0.0000	nan	0.0000
text/calendar	0.0332	0.0335	0.0370
text/css	0.0006	0.0004	0.0003
text/csv	0.0023	0.0023	0.0017
text/html	74.9337	76.1435	75.8900
text/plain	0.0609	0.0684	0.0618
text/tab-separated-values	0.0004	0.0004	0.0006
text/troff	0.0002	0.0003	0.0002
text/turtle	0.0004	0.0004	0.0004
text/x-chdr	0.0002	0.0005	0.0005
text/x-csrc	0.0006	0.0009	0.0007
text/x-diff	0.0011	0.0011	0.0012
text/x-jsp	0.0003	0.0003	0.0002
text/x-log	0.0015	0.0011	0.0019
text/x-matlab	0.0013	0.0020	0.0016
text/x-patch	0.0000	0.0000	0.0001
text/x-perl	0.0016	0.0021	0.0020
text/x-php	0.0055	0.0039	0.0039
text/x-vcalendar	0.0009	0.0008	0.0010
text/x-vcard	0.0030	0.0025	0.0029
video/mp4	0.0068	0.0011	0.0016
video/quicktime	0.0006	0.0004	0.0005
video/webm	0.0019	0.0001	0.0001
video/x-m4v	0.0007	0.0005	0.0006
video/x-matroska	0.0007	0.0008	0.0008
video/x-ms-wmv	0.0004	0.0001	0.0001