MIME Types

The crawled content is dominated by HTML pages and contains only a small percentage of other document formats. The tables show the percentage of the top 100 media or MIME types of the latest monthly crawls.

While the first table is based the Content-Type HTTP header, the second uses the MIME type detected by Apache Tika based on the actual content. The underlying data including page counts is provided in mimetypes.csv resp. mimetypes_detected.csv.

crawl	CC-MAIN-2021-17	CC-MAIN-2021-21	CC-MAIN-2021-25
mimetype
<other>	0.0656	0.0773	0.0832
application/atom+xml	0.0702	0.0724	0.0768
application/download	0.0025	0.0022	0.0024
application/epub+zip	0.0013	0.0013	0.0014
application/force-download	0.0039	0.0041	0.0041
application/javascript	0.0003	0.0005	0.0004
application/json	0.0164	0.0197	0.0216
application/ld+json	0.0007	0.0008	0.0006
application/marc	0.0004	0.0004	0.0004
application/msword	0.0026	0.0028	0.0025
application/octet-stream	0.0362	0.0439	0.0429
application/octetstream	0.0002	0.0003	0.0003
application/pdf	0.4576	1.0403	0.5308
application/pgp-encrypted	0.0000	0.0000	0.0000
application/pgp-signature	0.0008	0.0011	0.0021
application/postscript	0.0001	0.0002	0.0002
application/rdf+xml	0.0026	0.0026	0.0030
application/rss+xml	0.0625	0.0709	0.0606
application/rtf	0.0010	0.0011	0.0012
application/save-to-disk	0.0000	0.0000	nan
application/text	0.0021	0.0031	0.0022
application/unknown	0.0003	0.0012	0.0005
application/vnd.android.package-archive	0.0005	0.0006	0.0003
application/vnd.google-earth.kml+xml	0.0011	0.0016	0.0013
application/vnd.google-earth.kmz	0.0003	0.0004	0.0003
application/vnd.ms-excel	0.0021	0.0021	0.0024
application/vnd.ms-powerpoint	0.0009	0.0011	0.0012
application/vnd.ms-word	0.0004	0.0005	0.0004
application/vnd.ms-word.document.macroenabled.12	0.0000	0.0000	0.0000
application/vnd.oasis.opendocument.text	0.0008	0.0010	0.0010
application/vnd.openxmlformats-officedocument.presentationml.presentation	0.0002	0.0003	0.0002
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet	0.0011	0.0013	0.0013
application/vnd.openxmlformats-officedocument.wordprocessingml.document	0.0023	0.0027	0.0024
application/vnd.wap.xhtml+xml	0.0001	0.0001	0.0001
application/x-bibtex	0.0022	0.0021	0.0026
application/x-bittorrent	0.0002	0.0002	0.0002
application/x-debian-package	0.0001	0.0000	0.0000
application/x-download	0.0014	0.0008	0.0010
application/x-endnote-refer	0.0031	0.0041	0.0040
application/x-gzip	0.0008	0.0010	0.0011
application/x-httpd-php	0.0003	0.0007	0.0003
application/x-java-jnlp-file	0.0002	0.0004	0.0004
application/x-javascript	0.0004	0.0003	0.0003
application/x-json	0.0000	0.0000	0.0000
application/x-mobipocket-ebook	0.0002	0.0002	0.0002
application/x-msdownload	0.0003	0.0003	0.0003
application/x-netcdf	0.0007	0.0002	0.0004
application/x-rar-compressed	0.0001	0.0001	0.0001
application/x-research-info-systems	0.0060	0.0071	0.0078
application/x-shockwave-flash	0.0003	0.0004	0.0003
application/x-tar	0.0003	0.0003	0.0005
application/x-tex	0.0001	0.0002	0.0002
application/x-troff-man	0.0002	0.0003	0.0003
application/x-zip-compressed	0.0006	0.0006	0.0006
application/xhtml+xml	0.0469	0.0516	0.0543
application/xml	0.0282	0.0313	0.0296
application/zip	0.0040	0.0040	0.0044
audio/mp3	0.0002	0.0002	0.0002
audio/mp4	0.0000	0.0000	0.0000
audio/mpeg	0.0036	0.0035	0.0034
audio/x-mpegurl	0.0006	0.0009	0.0009
audio/x-pn-realaudio	0.0001	0.0001	0.0001
audio/x-scpls	0.0001	0.0002	0.0001
audio/x-wav	0.0001	0.0001	0.0001
binary/octet-stream	0.0007	0.0009	0.0007
image/gif	0.0045	0.0045	0.0045
image/jp2	0.0000	0.0000	0.0000
image/jpeg	0.1635	0.1487	0.1791
image/jpg	0.0024	0.0029	0.0030
image/pjpeg	0.0004	0.0005	0.0004
image/png	0.0332	0.0275	0.0367
image/svg+xml	0.0008	0.0007	0.0008
image/tiff	0.0004	0.0004	0.0004
image/vnd.djvu	0.0003	0.0002	0.0003
image/webp	0.0001	0.0003	0.0002
message/rfc822	0.0003	0.0004	0.0004
text/calendar	0.0247	0.0271	0.0257
text/css	0.0002	0.0004	0.0003
text/csv	0.0038	0.0041	0.0041
text/enriched	0.0008	0.0013	0.0008
text/html	98.7990	98.1682	98.6442
text/javascript	0.0003	0.0003	0.0003
text/pdf	0.0000	0.0000	0.0001
text/plain	0.0465	0.0589	0.0577
text/tab-separated-values	0.0004	0.0006	0.0005
text/turtle	0.0011	0.0009	0.0011
text/vcard	0.0009	0.0011	0.0009
text/vnd.wap.wml	0.0001	0.0001	0.0001
text/x-bibtex	0.0003	0.0006	0.0005
text/x-c	0.0001	0.0001	0.0002
text/x-csrc	0.0007	0.0006	0.0009
text/x-diff	0.0001	0.0004	0.0004
text/x-patch	0.0005	0.0006	0.0005
text/x-perl	0.0001	0.0001	0.0002
text/x-vcalendar	0.0004	0.0003	0.0003
text/x-vcard	0.0019	0.0021	0.0017
text/xml	0.0702	0.0729	0.0679
unknown/unknown	0.0005	0.0003	0.0002
video/mp4	0.0022	0.0022	0.0026
video/webm	0.0001	0.0001	0.0001
video/x-ms-asf	0.0001	0.0005	0.0004

crawl	CC-MAIN-2021-17	CC-MAIN-2021-21	CC-MAIN-2021-25
mimetype_detected
<other>	0.0130	0.0166	0.0155
application/atom+xml	0.0743	0.0773	0.0807
application/epub+zip	0.0016	0.0018	0.0018
application/gzip	0.0017	0.0018	0.0021
application/javascript	0.0008	0.0009	0.0009
application/json	0.0170	0.0196	0.0207
application/marc	0.0002	0.0001	0.0002
application/mbox	0.0033	0.0038	0.0035
application/msword	0.0021	0.0022	0.0020
application/octet-stream	0.0114	0.0165	0.0176
application/pdf	0.4722	1.0565	0.5442
application/pgp-signature	0.0020	0.0018	0.0028
application/pkcs7-signature	0.0003	0.0003	0.0004
application/pls+xml	0.0001	0.0002	0.0001
application/postscript	0.0001	0.0001	0.0001
application/rdf+xml	0.0072	0.0085	0.0079
application/rss+xml	0.1127	0.1253	0.1105
application/rtf	0.0017	0.0025	0.0019
application/text	0.0006	0.0006	0.0005
application/vnd.android.package-archive	0.0004	0.0004	0.0003
application/vnd.google-earth.kml+xml	0.0017	0.0022	0.0020
application/vnd.google-earth.kmz	0.0004	0.0004	0.0003
application/vnd.ms-excel	0.0017	0.0018	0.0021
application/vnd.ms-powerpoint	0.0008	0.0010	0.0011
application/vnd.oasis.opendocument.text	0.0007	0.0009	0.0007
application/vnd.openxmlformats-officedocument.presentationml.presentation	0.0003	0.0003	0.0002
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet	0.0011	0.0013	0.0012
application/vnd.openxmlformats-officedocument.wordprocessingml.document	0.0023	0.0026	0.0024
application/x-bibtex-text-file	0.0050	0.0061	0.0068
application/x-bittorrent	0.0002	0.0003	0.0003
application/x-bzip2	0.0008	0.0012	0.0009
application/x-debian-package	0.0001	0.0000	0.0000
application/x-dosexec	0.0004	0.0003	0.0003
application/x-endnote-refer	0.0017	0.0019	0.0019
application/x-mobipocket-ebook	0.0005	0.0005	0.0005
application/x-ms-asx	0.0001	0.0004	0.0003
application/x-msdownload	0.0002	0.0001	0.0002
application/x-rar-compressed	0.0000	0.0000	0.0000
application/x-research-info-systems	0.0001	0.0002	0.0002
application/x-sh	0.0009	0.0010	0.0010
application/x-shockwave-flash	0.0003	0.0004	0.0003
application/x-stata-do	0.0008	0.0005	0.0009
application/x-tika-msoffice	0.0026	0.0031	0.0028
application/x-tika-ooxml	0.0023	0.0022	0.0025
application/x-wais-source	0.0002	0.0006	0.0003
application/x-xz	0.0001	0.0001	0.0001
application/xhtml+xml	15.2434	14.9065	14.9836
application/xml	0.0712	0.0679	0.0669
application/zip	0.0075	0.0074	0.0084
application/zlib	0.0007	0.0007	0.0006
audio/midi	0.0002	0.0001	0.0001
audio/mp4	0.0001	0.0001	0.0000
audio/mpeg	0.0051	0.0052	0.0054
audio/vnd.wave	0.0004	0.0004	0.0005
audio/vorbis	0.0000	0.0000	0.0000
audio/x-mpegurl	0.0001	0.0001	0.0001
image/bmp	0.0001	0.0001	0.0001
image/gif	0.0046	0.0046	0.0046
image/jp2	0.0000	0.0000	0.0000
image/jpeg	0.1727	0.1587	0.1887
image/png	0.0330	0.0273	0.0370
image/svg+xml	0.0009	0.0007	0.0009
image/tiff	0.0004	0.0005	0.0005
image/vnd.djvu	0.0004	0.0002	0.0006
image/vnd.dxf; format=ascii	0.0003	0.0004	0.0004
image/vnd.microsoft.icon	0.0001	0.0002	0.0002
image/webp	0.0001	0.0003	0.0003
message/rfc822	0.0018	0.0030	0.0026
text/calendar	0.0265	0.0289	0.0275
text/css	0.0002	0.0004	0.0003
text/csv	0.0037	0.0041	0.0039
text/html	83.5884	83.3000	83.7088
text/plain	0.0652	0.0843	0.0858
text/prs.lines.tag	0.0007	0.0009	0.0010
text/tab-separated-values	0.0004	0.0006	0.0005
text/troff	0.0003	0.0003	0.0004
text/turtle	0.0010	0.0009	0.0011
text/vtt	0.0006	0.0008	0.0008
text/x-c++src	0.0004	0.0004	0.0003
text/x-chdr	0.0008	0.0008	0.0009
text/x-csrc	0.0016	0.0016	0.0022
text/x-diff	0.0021	0.0027	0.0027
text/x-jsp	0.0001	0.0003	0.0001
text/x-log	0.0017	0.0025	0.0024
text/x-matlab	0.0020	0.0025	0.0026
text/x-patch	0.0002	0.0003	0.0002
text/x-perl	0.0025	0.0030	0.0027
text/x-php	0.0034	0.0059	0.0036
text/x-python	0.0004	0.0006	0.0005
text/x-vcalendar	0.0005	0.0004	0.0004
text/x-vcard	0.0034	0.0040	0.0032
video/mp4	0.0023	0.0023	0.0027
video/quicktime	0.0003	0.0002	0.0003
video/webm	0.0001	0.0001	0.0001
video/x-m4v	0.0000	0.0000	0.0000
video/x-matroska	0.0000	0.0000	0.0000
video/x-ms-wmv	0.0001	0.0001	0.0001