Character Encoding of HTML Pages

The character set or encoding of HTML pages only is identified by Tika’s AutoDetectReader. The table shows the percentage how character sets have been used to encode HTML pages crawled by the latest monthly crawls. The underlying data including page counts is provided in charsets.csv.

crawl	CC-MAIN-2024-46	CC-MAIN-2024-51	CC-MAIN-2025-05
charset
<other>	0.0001	0.0001	0.0001
<unknown>	1.7604	1.7906	1.4664
Big5	0.0856	0.0458	0.0463
Big5-HKSCS	0.0002	0.0000	0.0000
EUC-JP	0.1199	0.1279	0.1181
EUC-KR	0.0878	0.0917	0.0839
GB18030	0.0150	0.0152	0.0129
GB2312	0.2362	0.2477	0.2164
GBK	0.1114	0.1194	0.1015
IBM420	0.0045	0.0063	0.0041
IBM424	0.0016	0.0017	0.0014
IBM500	0.0008	0.0012	0.0013
IBM855	0.0000	0.0000	0.0000
IBM866	0.0003	0.0003	0.0003
ISO-2022-JP	0.0011	0.0012	0.0009
ISO-8859-1	2.2498	2.4097	2.2951
ISO-8859-13	0.0001	0.0001	0.0001
ISO-8859-15	0.0471	0.0496	0.0433
ISO-8859-16	0.0002	0.0002	0.0002
ISO-8859-2	0.1036	0.1165	0.1059
ISO-8859-3	0.0009	0.0005	0.0001
ISO-8859-4	0.0007	0.0008	0.0008
ISO-8859-5	0.0010	0.0012	0.0012
ISO-8859-6	0.0000	0.0000	0.0000
ISO-8859-7	0.0058	0.0065	0.0060
ISO-8859-8	0.0008	0.0008	0.0008
ISO-8859-9	0.0227	0.0221	0.0228
KOI8-R	0.0075	0.0061	0.0057
KOI8-U	0.0001	0.0001	0.0000
Shift_JIS	0.1786	0.1821	0.1589
TIS-620	0.0063	0.0056	0.0047
US-ASCII	0.0354	0.0407	0.0314
UTF-16	0.0054	0.0055	0.0051
UTF-16BE	0.0002	0.0002	0.0002
UTF-16LE	0.0012	0.0011	0.0010
UTF-32	0.0000	0.0000	0.0000
UTF-32LE	0.0002	0.0001	0.0003
UTF-8	94.1026	93.8442	94.4849
windows-1250	0.0732	0.0734	0.0677
windows-1251	0.5011	0.5437	0.4926
windows-1252	0.1590	0.1628	0.1395
windows-1253	0.0026	0.0027	0.0025
windows-1254	0.0114	0.0127	0.0123
windows-1255	0.0072	0.0076	0.0074
windows-1256	0.0370	0.0392	0.0404
windows-1257	0.0059	0.0064	0.0071
windows-1258	0.0000	0.0000	0.0000
windows-31j	0.0005	0.0005	0.0005
x-windows-874	0.0071	0.0082	0.0082
x-windows-949	0.0000	0.0000	0.0000