Character Encoding of HTML Pages

The character set or encoding of HTML pages only is identified by Tika’s AutoDetectReader. The table shows the percentage how character sets have been used to encode HTML pages crawled by the latest monthly crawls. The underlying data including page counts is provided in charsets.csv.

crawl	CC-MAIN-2024-38	CC-MAIN-2024-42	CC-MAIN-2024-46
charset
<other>	0.0000	0.0001	0.0001
<unknown>	1.5057	2.0343	1.7604
Big5	0.0823	0.0861	0.0856
Big5-HKSCS	0.0001	0.0001	0.0002
EUC-JP	0.1006	0.1204	0.1199
EUC-KR	0.0837	0.0906	0.0878
GB18030	0.0150	0.0142	0.0150
GB2312	0.2230	0.2367	0.2362
GBK	0.1110	0.1094	0.1114
IBM420	0.0040	0.0047	0.0045
IBM424	0.0014	0.0019	0.0016
IBM500	0.0007	0.0006	0.0008
IBM855	0.0000	0.0000	0.0000
IBM866	0.0003	0.0003	0.0003
ISO-2022-JP	0.0009	0.0010	0.0011
ISO-8859-1	2.7588	2.6607	2.2498
ISO-8859-13	0.0001	0.0002	0.0001
ISO-8859-15	0.0466	0.0516	0.0471
ISO-8859-16	0.0001	0.0001	0.0002
ISO-8859-2	0.1072	0.1148	0.1036
ISO-8859-3	0.0005	0.0013	0.0009
ISO-8859-4	0.0008	0.0007	0.0007
ISO-8859-5	0.0019	0.0017	0.0010
ISO-8859-6	0.0001	0.0000	0.0000
ISO-8859-7	0.0054	0.0064	0.0058
ISO-8859-8	0.0008	0.0008	0.0008
ISO-8859-9	0.0235	0.0244	0.0227
KOI8-R	0.0065	0.0079	0.0075
KOI8-U	0.0001	0.0001	0.0001
Shift_JIS	0.1602	0.1888	0.1786
TIS-620	0.0054	0.0062	0.0063
US-ASCII	0.0338	0.0355	0.0354
UTF-16	0.0047	0.0058	0.0054
UTF-16BE	0.0003	0.0004	0.0002
UTF-16LE	0.0011	0.0014	0.0012
UTF-32	0.0000	0.0001	0.0000
UTF-32LE	0.0002	0.0002	0.0002
UTF-8	93.9167	93.3348	94.1026
windows-1250	0.0696	0.0785	0.0732
windows-1251	0.4984	0.5192	0.5011
windows-1252	0.1552	0.1798	0.1590
windows-1253	0.0022	0.0029	0.0026
windows-1254	0.0111	0.0121	0.0114
windows-1255	0.0067	0.0073	0.0072
windows-1256	0.0380	0.0403	0.0370
windows-1257	0.0075	0.0075	0.0059
windows-1258	0.0000	0.0000	0.0000
windows-31j	0.0006	0.0006	0.0005
x-windows-874	0.0070	0.0075	0.0071
x-windows-949	0.0000	0.0000	0.0000