Character Encoding of HTML Pages

The character set or encoding of HTML pages only is identified by Tika’s AutoDetectReader. The table shows the percentage how character sets have been used to encode HTML pages crawled by the latest monthly crawls. The underlying data including page counts is provided in charsets.csv.

crawl	CC-MAIN-2024-42	CC-MAIN-2024-46	CC-MAIN-2024-51
charset
<other>	0.0001	0.0001	0.0001
<unknown>	2.0343	1.7604	1.7906
Big5	0.0861	0.0856	0.0458
Big5-HKSCS	0.0001	0.0002	0.0000
EUC-JP	0.1204	0.1199	0.1279
EUC-KR	0.0906	0.0878	0.0917
GB18030	0.0142	0.0150	0.0152
GB2312	0.2367	0.2362	0.2477
GBK	0.1094	0.1114	0.1194
IBM420	0.0047	0.0045	0.0063
IBM424	0.0019	0.0016	0.0017
IBM500	0.0006	0.0008	0.0012
IBM855	0.0000	0.0000	0.0000
IBM866	0.0003	0.0003	0.0003
ISO-2022-JP	0.0010	0.0011	0.0012
ISO-8859-1	2.6607	2.2498	2.4097
ISO-8859-13	0.0002	0.0001	0.0001
ISO-8859-15	0.0516	0.0471	0.0496
ISO-8859-16	0.0001	0.0002	0.0002
ISO-8859-2	0.1148	0.1036	0.1165
ISO-8859-3	0.0013	0.0009	0.0005
ISO-8859-4	0.0007	0.0007	0.0008
ISO-8859-5	0.0017	0.0010	0.0012
ISO-8859-6	0.0000	0.0000	0.0000
ISO-8859-7	0.0064	0.0058	0.0065
ISO-8859-8	0.0008	0.0008	0.0008
ISO-8859-9	0.0244	0.0227	0.0221
KOI8-R	0.0079	0.0075	0.0061
KOI8-U	0.0001	0.0001	0.0001
Shift_JIS	0.1888	0.1786	0.1821
TIS-620	0.0062	0.0063	0.0056
US-ASCII	0.0355	0.0354	0.0407
UTF-16	0.0058	0.0054	0.0055
UTF-16BE	0.0004	0.0002	0.0002
UTF-16LE	0.0014	0.0012	0.0011
UTF-32	0.0001	0.0000	0.0000
UTF-32LE	0.0002	0.0002	0.0001
UTF-8	93.3348	94.1026	93.8442
windows-1250	0.0785	0.0732	0.0734
windows-1251	0.5192	0.5011	0.5437
windows-1252	0.1798	0.1590	0.1628
windows-1253	0.0029	0.0026	0.0027
windows-1254	0.0121	0.0114	0.0127
windows-1255	0.0073	0.0072	0.0076
windows-1256	0.0403	0.0370	0.0392
windows-1257	0.0075	0.0059	0.0064
windows-1258	0.0000	0.0000	0.0000
windows-31j	0.0006	0.0005	0.0005
x-windows-874	0.0075	0.0071	0.0082
x-windows-949	0.0000	0.0000	0.0000