Statistics of Common Crawl Monthly Archives

Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives
Latest crawl: CC-MAIN-2024-22

View the Project on GitHub

Character Encoding of HTML Pages

The character set or encoding of HTML pages only is identified by Tika’s AutoDetectReader. The table shows the percentage how character sets have been used to encode HTML pages crawled by the latest monthly crawls. The underlying data including page counts is provided in charsets.csv.

crawl CC-MAIN-2024-10 CC-MAIN-2024-18 CC-MAIN-2024-22
charset
<other> 0.0000 0.0001 0.0000
<unknown> 1.6892 1.7163 1.6020
Big5 0.0554 0.0364 0.0432
Big5-HKSCS 0.0000 0.0000 0.0000
EUC-JP 0.1060 0.1091 0.1013
EUC-KR 0.0806 0.0830 0.0777
GB18030 0.0157 0.0167 0.0146
GB2312 0.2435 0.2712 0.2566
GBK 0.1052 0.1083 0.0959
IBM420 0.0056 0.0055 0.0052
IBM424 0.0022 0.0023 0.0021
IBM500 0.0007 0.0007 0.0008
IBM855 0.0000 0.0000 0.0000
IBM866 0.0001 0.0001 0.0003
ISO-2022-JP 0.0012 0.0014 0.0011
ISO-8859-1 2.3258 2.2744 2.2593
ISO-8859-13 0.0000 0.0000 0.0001
ISO-8859-15 0.0500 0.0484 0.0484
ISO-8859-16 0.0001 0.0001 0.0001
ISO-8859-2 0.1261 0.1140 0.1132
ISO-8859-3 0.0005 0.0005 0.0005
ISO-8859-4 0.0007 0.0007 0.0007
ISO-8859-5 0.0026 0.0028 0.0029
ISO-8859-6 0.0001 0.0000 0.0000
ISO-8859-7 0.0095 0.0069 0.0064
ISO-8859-8 0.0009 0.0007 0.0008
ISO-8859-9 0.0261 0.0258 0.0239
KOI8-R 0.0077 0.0072 0.0075
KOI8-U 0.0000 0.0001 0.0001
Shift_JIS 0.1865 0.1905 0.1773
TIS-620 0.0051 0.0053 0.0053
US-ASCII 0.0349 0.0352 0.0338
UTF-16 0.0034 0.0037 0.0042
UTF-16BE 0.0003 0.0003 0.0002
UTF-16LE 0.0015 0.0016 0.0014
UTF-32 0.0000 0.0000 0.0000
UTF-32LE 0.0003 0.0003 0.0002
UTF-8 94.0329 94.0871 94.2852
windows-1250 0.0748 0.0729 0.0765
windows-1251 0.5250 0.5091 0.5012
windows-1252 0.1836 0.1804 0.1694
windows-1253 0.0026 0.0027 0.0026
windows-1254 0.0095 0.0114 0.0107
windows-1255 0.0076 0.0071 0.0066
windows-1256 0.0538 0.0412 0.0431
windows-1257 0.0125 0.0079 0.0085
windows-1258 0.0000 0.0000 0.0000
windows-31j 0.0005 0.0005 0.0005
x-windows-874 0.0097 0.0101 0.0085
x-windows-949 0.0001 0.0000 0.0000