Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives
The crawled content is dominated by HTML pages and contains only a small percentage of other document formats. The table shows the percentage of the top 100 media or MIME types of the latest monthly crawls.
| crawl | CC-MAIN-2018-43 | CC-MAIN-2018-47 | CC-MAIN-2018-51 |
|---|---|---|---|
| mimetype | |||
| <other> | 0.6520 | 0.6122 | 0.5519 |
| application/atom+xml | 0.0833 | 0.0948 | 0.0817 |
| application/download | 0.0014 | 0.0016 | 0.0014 |
| application/epub+zip | 0.0006 | 0.0008 | 0.0008 |
| application/force-download | 0.0029 | 0.0034 | 0.0033 |
| application/javascript | 0.0003 | 0.0005 | 0.0003 |
| application/json | 0.0099 | 0.0093 | 0.0094 |
| application/jsonrequest | 0.0000 | 0.0000 | 0.0000 |
| application/ld+json | 0.0002 | 0.0003 | 0.0001 |
| application/marc | 0.0000 | 0.0000 | 0.0000 |
| application/msword | 0.0048 | 0.0109 | 0.0042 |
| application/octet-stream | 0.0283 | 0.0458 | 0.0328 |
| application/octetstream | 0.0002 | 0.0002 | 0.0003 |
| application/pdf | 0.8036 | 0.9513 | 1.1398 |
| application/pgp-encrypted | 0.0000 | 0.0000 | 0.0000 |
| application/pgp-signature | 0.0005 | 0.0021 | 0.0005 |
| application/postscript | 0.0007 | 0.0011 | 0.0010 |
| application/rdf+xml | 0.0017 | 0.0023 | 0.0017 |
| application/rss+xml | 0.0721 | 0.0711 | 0.0795 |
| application/save-to-disk | 0.0000 | nan | 0.0000 |
| application/text | 0.0003 | 0.0003 | 0.0003 |
| application/unknown | 0.0003 | 0.0003 | 0.0003 |
| application/vnd.android.package-archive | 0.0008 | 0.0017 | 0.0011 |
| application/vnd.google-earth.kml+xml | 0.0010 | 0.0017 | 0.0015 |
| application/vnd.google-earth.kmz | 0.0003 | 0.0005 | 0.0004 |
| application/vnd.ms-excel | 0.0016 | 0.0033 | 0.0014 |
| application/vnd.ms-powerpoint | 0.0004 | 0.0011 | 0.0004 |
| application/vnd.ms-word | 0.0002 | 0.0001 | 0.0001 |
| application/vnd.ms-word.document.macroenabled.12 | 0.0003 | 0.0031 | 0.0013 |
| application/vnd.openxmlformats-officedocument.presentationml.presentation | 0.0005 | 0.0003 | 0.0003 |
| application/vnd.openxmlformats-officedocument.spreadsheetml.sheet | 0.0013 | 0.0010 | 0.0009 |
| application/vnd.openxmlformats-officedocument.wordprocessingml.document | 0.0038 | 0.0022 | 0.0023 |
| application/vnd.wap.xhtml+xml | 0.0003 | 0.0001 | 0.0002 |
| application/x-bibtex | 0.0004 | 0.0004 | 0.0003 |
| application/x-bittorrent | 0.0005 | 0.0005 | 0.0004 |
| application/x-debian-package | 0.0002 | 0.0001 | 0.0000 |
| application/x-download | 0.0007 | 0.0009 | 0.0006 |
| application/x-dvi | 0.0001 | 0.0001 | 0.0000 |
| application/x-endnote-refer | 0.0005 | 0.0006 | 0.0004 |
| application/x-gzip | 0.0010 | 0.0009 | 0.0007 |
| application/x-httpd-php | 0.0007 | 0.0008 | 0.0007 |
| application/x-java-archive | 0.0000 | 0.0000 | 0.0000 |
| application/x-java-jnlp-file | 0.0001 | 0.0001 | 0.0001 |
| application/x-javascript | 0.0002 | 0.0002 | 0.0002 |
| application/x-json | 0.0000 | 0.0000 | 0.0000 |
| application/x-mobipocket-ebook | 0.0001 | 0.0002 | 0.0002 |
| application/x-msdownload | 0.0004 | 0.0003 | 0.0003 |
| application/x-netcdf | 0.0000 | 0.0001 | 0.0000 |
| application/x-research-info-systems | 0.0008 | 0.0010 | 0.0008 |
| application/x-shockwave-flash | 0.0016 | 0.0021 | 0.0015 |
| application/x-tar | 0.0002 | 0.0003 | 0.0002 |
| application/x-tex | 0.0001 | 0.0001 | 0.0001 |
| application/x-troff-man | 0.0001 | 0.0001 | 0.0001 |
| application/x-xz | 0.0003 | 0.0017 | 0.0006 |
| application/x-zip-compressed | 0.0003 | 0.0007 | 0.0005 |
| application/xhtml+xml | 0.0210 | 0.0170 | 0.0208 |
| application/xml | 0.0279 | 0.0264 | 0.0262 |
| application/zip | 0.0035 | 0.0087 | 0.0035 |
| audio/mp3 | 0.0001 | 0.0002 | 0.0003 |
| audio/mpeg | 0.0050 | 0.0065 | 0.0056 |
| audio/x-mpegurl | 0.0011 | 0.0019 | 0.0014 |
| audio/x-pn-realaudio | 0.0002 | 0.0001 | 0.0001 |
| audio/x-scpls | 0.0001 | 0.0002 | 0.0002 |
| audio/x-wav | 0.0004 | 0.0003 | 0.0003 |
| binary/octet-stream | 0.0004 | 0.0004 | 0.0006 |
| image/gif | 0.0052 | 0.0066 | 0.0044 |
| image/jp2 | 0.0000 | 0.0000 | 0.0000 |
| image/jpeg | 0.2375 | 0.3043 | 0.2732 |
| image/jpg | 0.0021 | 0.0028 | 0.0024 |
| image/pjpeg | 0.0002 | 0.0003 | 0.0003 |
| image/png | 0.0205 | 0.0317 | 0.0231 |
| image/svg+xml | 0.0029 | 0.0024 | 0.0013 |
| image/tiff | 0.0004 | 0.0004 | 0.0003 |
| image/vnd.djvu | 0.0003 | 0.0006 | 0.0004 |
| image/webp | 0.0008 | 0.0004 | 0.0005 |
| message/rfc822 | 0.0002 | 0.0004 | 0.0002 |
| text/calendar | 0.0176 | 0.0136 | 0.0207 |
| text/csv | 0.0005 | 0.0005 | 0.0005 |
| text/enriched | 0.0001 | 0.0001 | 0.0001 |
| text/html | 97.8543 | 97.6118 | 97.5659 |
| text/javascript | 0.0003 | 0.0003 | 0.0003 |
| text/pdf | 0.0000 | 0.0000 | 0.0000 |
| text/plain | 0.0263 | 0.0445 | 0.0317 |
| text/tab-separated-values | 0.0001 | 0.0001 | 0.0001 |
| text/turtle | 0.0001 | 0.0001 | 0.0001 |
| text/vnd.wap.wml | 0.0003 | 0.0005 | 0.0002 |
| text/x-bibtex | 0.0002 | 0.0002 | 0.0003 |
| text/x-c | 0.0001 | 0.0002 | 0.0001 |
| text/x-csrc | 0.0001 | 0.0001 | 0.0001 |
| text/x-diff | 0.0002 | 0.0002 | 0.0002 |
| text/x-patch | 0.0000 | 0.0000 | 0.0000 |
| text/x-perl | 0.0001 | 0.0003 | 0.0001 |
| text/x-vcalendar | 0.0004 | 0.0003 | 0.0005 |
| text/x-vcard | 0.0008 | 0.0010 | 0.0019 |
| text/xml | 0.0820 | 0.0731 | 0.0791 |
| unknown/unknown | 0.0002 | 0.0002 | 0.0001 |
| video/mp4 | 0.0020 | 0.0023 | 0.0018 |
| video/webm | 0.0020 | 0.0028 | 0.0033 |
| video/x-ms-asf | 0.0004 | 0.0010 | 0.0006 |
| video/x-ms-wmv | 0.0005 | 0.0003 | 0.0002 |
| video/x-ms-wvx | 0.0000 | 0.0000 | 0.0000 |