MIME Types

The crawled content is dominated by HTML pages and contains only a small percentage of other document formats. The tables show the percentage of the top 100 media or MIME types of the latest monthly crawls.

While the first table is based the Content-Type HTTP header, the second uses the MIME type detected by Apache Tika based on the actual content. The underlying data including page counts is provided in mimetypes.csv resp. mimetypes_detected.csv.

crawl	CC-MAIN-2020-05	CC-MAIN-2020-10	CC-MAIN-2020-16
mimetype
<other>	0.7258	0.8442	0.8413
application/atom+xml	0.1232	0.1366	0.0917
application/download	0.0021	0.0032	0.0023
application/epub+zip	0.0010	0.0018	0.0013
application/force-download	0.0044	0.0068	0.0042
application/javascript	0.0004	0.0006	0.0004
application/json	0.0189	0.0226	0.0166
application/ld+json	0.0003	0.0005	0.0003
application/marc	0.0004	0.0005	0.0004
application/msword	0.0020	0.0034	0.0027
application/octet-stream	0.0352	0.0478	0.0364
application/octetstream	0.0001	0.0002	0.0002
application/pdf	0.0792	0.1301	0.7246
application/pgp-encrypted	0.0000	0.0000	0.0000
application/pgp-signature	0.0009	0.0013	0.0013
application/postscript	0.0001	0.0002	0.0002
application/rdf+xml	0.0023	0.0031	0.0022
application/rss+xml	0.0726	0.0844	0.0717
application/rtf	0.0009	0.0014	0.0011
application/save-to-disk	0.0000	0.0000	0.0000
application/text	0.0027	0.0029	0.0022
application/unknown	0.0003	0.0003	0.0003
application/vnd.android.package-archive	0.0008	0.0015	0.0010
application/vnd.google-earth.kml+xml	0.0016	0.0019	0.0013
application/vnd.google-earth.kmz	0.0003	0.0005	0.0003
application/vnd.ms-excel	0.0011	0.0018	0.0014
application/vnd.ms-powerpoint	0.0007	0.0009	0.0007
application/vnd.ms-word	0.0006	0.0007	0.0005
application/vnd.ms-word.document.macroenabled.12	0.0000	0.0000	0.0000
application/vnd.oasis.opendocument.text	0.0006	0.0010	0.0008
application/vnd.openxmlformats-officedocument.presentationml.presentation	0.0002	0.0004	0.0002
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet	0.0010	0.0014	0.0010
application/vnd.openxmlformats-officedocument.wordprocessingml.document	0.0018	0.0030	0.0020
application/vnd.wap.xhtml+xml	0.0001	0.0001	0.0001
application/x-bibtex	0.0017	0.0023	0.0015
application/x-bittorrent	0.0003	0.0005	0.0003
application/x-debian-package	0.0001	0.0001	0.0003
application/x-download	0.0010	0.0015	0.0012
application/x-endnote-refer	0.0032	0.0038	0.0027
application/x-gzip	0.0008	0.0010	0.0009
application/x-httpd-php	0.0006	0.0008	0.0005
application/x-java-jnlp-file	0.0003	0.0002	0.0001
application/x-javascript	0.0004	0.0006	0.0002
application/x-json	0.0000	0.0000	0.0000
application/x-mobipocket-ebook	0.0001	0.0001	0.0001
application/x-msdownload	0.0003	0.0004	0.0003
application/x-netcdf	0.0000	0.0003	0.0001
application/x-research-info-systems	0.0057	0.0071	0.0053
application/x-shockwave-flash	0.0005	0.0006	0.0003
application/x-tar	0.0002	0.0004	0.0002
application/x-tex	0.0001	0.0001	0.0001
application/x-troff-man	0.0003	0.0002	0.0003
application/x-xz	0.0000	0.0001	0.0001
application/x-zip-compressed	0.0005	0.0009	0.0006
application/xhtml+xml	0.0374	0.0455	0.0359
application/xml	0.0290	0.0340	0.0292
application/zip	0.0035	0.0048	0.0037
audio/mp3	0.0002	0.0004	0.0002
audio/mp4	0.0004	0.0008	0.0006
audio/mpeg	0.0029	0.0053	0.0043
audio/x-mpegurl	0.0010	0.0015	0.0008
audio/x-pn-realaudio	0.0001	0.0001	0.0001
audio/x-scpls	0.0001	0.0002	0.0001
audio/x-wav	0.0006	0.0005	0.0004
binary/octet-stream	0.0006	0.0007	0.0007
image/gif	0.0024	0.0032	0.0024
image/jp2	0.0002	0.0002	0.0001
image/jpeg	0.0963	0.1456	0.1211
image/jpg	0.0020	0.0032	0.0026
image/pjpeg	0.0006	0.0005	0.0004
image/png	0.0142	0.0207	0.0198
image/svg+xml	0.0003	0.0004	0.0004
image/tiff	0.0003	0.0003	0.0002
image/vnd.djvu	0.0003	0.0006	0.0003
image/webp	0.0005	0.0009	0.0007
message/rfc822	0.0004	0.0007	0.0004
text/calendar	0.0342	0.0483	0.0311
text/csv	0.0026	0.0036	0.0027
text/directory	0.0004	0.0005	0.0004
text/enriched	0.0010	0.0010	0.0007
text/html	98.5013	98.1498	97.7633
text/javascript	0.0006	0.0005	0.0005
text/pdf	0.0000	0.0000	0.0000
text/plain	0.0720	0.0755	0.0514
text/tab-separated-values	0.0005	0.0007	0.0004
text/turtle	0.0008	0.0010	0.0009
text/vcard	0.0009	0.0012	0.0008
text/x-bibtex	0.0004	0.0005	0.0003
text/x-c	0.0001	0.0001	0.0001
text/x-csrc	0.0007	0.0010	0.0007
text/x-diff	0.0002	0.0004	0.0002
text/x-patch	0.0004	0.0005	0.0004
text/x-perl	0.0001	0.0002	0.0002
text/x-vcalendar	0.0005	0.0010	0.0006
text/x-vcard	0.0021	0.0030	0.0017
text/xml	0.0880	0.1125	0.0915
unknown/unknown	0.0006	0.0006	0.0006
video/mp4	0.0010	0.0014	0.0012
video/webm	0.0001	0.0001	0.0001
video/x-ms-asf	0.0004	0.0004	0.0002

crawl	CC-MAIN-2020-05	CC-MAIN-2020-10	CC-MAIN-2020-16
mimetype_detected
<other>	0.0117	0.0163	0.0126
application/atom+xml	0.1280	0.1421	0.0966
application/epub+zip	0.0013	0.0022	0.0016
application/force-download	0.0002	0.0002	0.0001
application/gzip	0.0015	0.0020	0.0025
application/javascript	0.0012	0.0014	0.0011
application/json	0.0197	0.0234	0.0164
application/marc	0.0002	0.0003	0.0002
application/mbox	0.0022	0.0029	0.0019
application/msword	0.0015	0.0027	0.0022
application/octet-stream	0.0105	0.0170	0.0092
application/pdf	0.0876	0.1441	0.7425
application/pgp-signature	0.0010	0.0017	0.0016
application/pls+xml	0.0001	0.0002	0.0002
application/postscript	0.0001	0.0002	0.0001
application/rdf+xml	0.0096	0.0116	0.0100
application/rss+xml	0.1307	0.1546	0.1300
application/rtf	0.0022	0.0028	0.0021
application/text	0.0005	0.0006	0.0005
application/vnd.android.package-archive	0.0010	0.0019	0.0014
application/vnd.google-earth.kml+xml	0.0020	0.0025	0.0017
application/vnd.google-earth.kmz	0.0004	0.0005	0.0003
application/vnd.ms-excel	0.0008	0.0014	0.0011
application/vnd.ms-powerpoint	0.0007	0.0009	0.0008
application/vnd.oasis.opendocument.text	0.0006	0.0011	0.0006
application/vnd.openxmlformats-officedocument.presentationml.presentation	0.0002	0.0004	0.0002
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet	0.0010	0.0014	0.0010
application/vnd.openxmlformats-officedocument.wordprocessingml.document	0.0018	0.0031	0.0021
application/x-bibtex-text-file	0.0051	0.0068	0.0050
application/x-bittorrent	0.0003	0.0006	0.0003
application/x-bzip2	0.0003	0.0007	0.0002
application/x-debian-package	0.0002	0.0001	0.0004
application/x-dosexec	0.0004	0.0005	0.0004
application/x-endnote-refer	0.0017	0.0023	0.0015
application/x-mobipocket-ebook	0.0003	0.0004	0.0003
application/x-ms-asx	0.0003	0.0004	0.0002
application/x-msdownload	0.0001	0.0002	0.0001
application/x-rar-compressed	0.0004	0.0007	0.0005
application/x-research-info-systems	0.0002	0.0002	0.0002
application/x-sh	0.0009	0.0012	0.0009
application/x-shockwave-flash	0.0005	0.0006	0.0003
application/x-stata-do	0.0004	0.0003	0.0002
application/x-tika-msoffice	0.0025	0.0042	0.0025
application/x-tika-ooxml	0.0017	0.0027	0.0019
application/x-wais-source	0.0002	0.0003	0.0002
application/x-xz	0.0002	0.0003	0.0002
application/xhtml+xml	21.0508	21.4964	20.1664
application/xml	0.0794	0.1011	0.0842
application/zip	0.0065	0.0092	0.0076
application/zlib	0.0003	0.0002	0.0002
audio/midi	0.0001	0.0001	0.0001
audio/mp4	0.0009	0.0017	0.0015
audio/mpeg	0.0043	0.0070	0.0056
audio/vnd.wave	0.0008	0.0012	0.0008
audio/vorbis	0.0001	0.0001	0.0001
audio/x-mpegurl	0.0001	0.0002	0.0001
image/bmp	0.0001	0.0001	0.0001
image/gif	0.0024	0.0032	0.0025
image/jp2	0.0002	0.0002	0.0001
image/jpeg	0.1039	0.1576	0.1310
image/png	0.0152	0.0218	0.0209
image/svg+xml	0.0004	0.0006	0.0005
image/tiff	0.0005	0.0005	0.0003
image/vnd.djvu	0.0005	0.0009	0.0006
image/vnd.dxf; format=ascii	0.0003	0.0006	0.0003
image/vnd.microsoft.icon	0.0002	0.0004	0.0002
image/webp	0.0006	0.0010	0.0009
message/rfc822	0.0013	0.0021	0.0013
text/calendar	0.0357	0.0508	0.0328
text/css	0.0003	0.0003	0.0003
text/csv	0.0024	0.0034	0.0026
text/html	78.1387	77.4394	78.3861
text/plain	0.0930	0.1037	0.0729
text/prs.lines.tag	0.0015	0.0009	0.0011
text/tab-separated-values	0.0005	0.0008	0.0004
text/troff	0.0003	0.0003	0.0003
text/turtle	0.0008	0.0010	0.0009
text/vtt	0.0004	0.0006	0.0005
text/x-c++src	0.0003	0.0004	0.0003
text/x-chdr	0.0010	0.0012	0.0010
text/x-csrc	0.0017	0.0020	0.0015
text/x-diff	0.0019	0.0025	0.0019
text/x-jsp	0.0002	0.0004	0.0001
text/x-log	0.0024	0.0040	0.0015
text/x-matlab	0.0018	0.0024	0.0015
text/x-patch	0.0001	0.0002	0.0002
text/x-perl	0.0039	0.0043	0.0031
text/x-php	0.0035	0.0038	0.0038
text/x-python	0.0003	0.0004	0.0003
text/x-vcalendar	0.0006	0.0011	0.0007
text/x-vcard	0.0038	0.0054	0.0033
video/mp4	0.0012	0.0017	0.0014
video/quicktime	0.0002	0.0005	0.0004
video/webm	0.0001	0.0001	0.0001
video/x-m4v	0.0001	0.0001	0.0001
video/x-matroska	0.0002	0.0003	0.0002
video/x-ms-wmv	0.0001	0.0001	0.0001