Character Encoding of HTML Pages

The character set or encoding of HTML pages only is identified by Tika’s AutoDetectReader. The table shows the percentage how character sets have been used to encode HTML pages crawled by the latest monthly crawls. The underlying data including page counts is provided in charsets.csv.

crawl	CC-MAIN-2024-18	CC-MAIN-2024-22	CC-MAIN-2024-26
charset
<other>	0.0001	0.0000	0.0001
<unknown>	1.7163	1.6020	1.6322
Big5	0.0364	0.0432	0.0740
Big5-HKSCS	0.0000	0.0000	0.0000
EUC-JP	0.1091	0.1013	0.1076
EUC-KR	0.0830	0.0777	0.1034
GB18030	0.0167	0.0146	0.0178
GB2312	0.2712	0.2566	0.4719
GBK	0.1083	0.0959	0.1341
IBM420	0.0055	0.0052	0.0052
IBM424	0.0023	0.0021	0.0020
IBM500	0.0007	0.0008	0.0007
IBM855	0.0000	0.0000	0.0000
IBM866	0.0001	0.0003	0.0002
ISO-2022-JP	0.0014	0.0011	0.0011
ISO-8859-1	2.2744	2.2593	2.3257
ISO-8859-13	0.0000	0.0001	0.0001
ISO-8859-15	0.0484	0.0484	0.0487
ISO-8859-16	0.0001	0.0001	0.0001
ISO-8859-2	0.1140	0.1132	0.1164
ISO-8859-3	0.0005	0.0005	0.0004
ISO-8859-4	0.0007	0.0007	0.0007
ISO-8859-5	0.0028	0.0029	0.0031
ISO-8859-6	0.0000	0.0000	0.0000
ISO-8859-7	0.0069	0.0064	0.0071
ISO-8859-8	0.0007	0.0008	0.0006
ISO-8859-9	0.0258	0.0239	0.0247
KOI8-R	0.0072	0.0075	0.0072
KOI8-U	0.0001	0.0001	0.0001
Shift_JIS	0.1905	0.1773	0.1780
TIS-620	0.0053	0.0053	0.0060
US-ASCII	0.0352	0.0338	0.0327
UTF-16	0.0037	0.0042	0.0049
UTF-16BE	0.0003	0.0002	0.0004
UTF-16LE	0.0016	0.0014	0.0015
UTF-32	0.0000	0.0000	0.0000
UTF-32LE	0.0003	0.0002	0.0002
UTF-8	94.0871	94.2852	93.8361
windows-1250	0.0729	0.0765	0.0782
windows-1251	0.5091	0.5012	0.5168
windows-1252	0.1804	0.1694	0.1721
windows-1253	0.0027	0.0026	0.0026
windows-1254	0.0114	0.0107	0.0114
windows-1255	0.0071	0.0066	0.0062
windows-1256	0.0412	0.0431	0.0487
windows-1257	0.0079	0.0085	0.0096
windows-1258	0.0000	0.0000	0.0000
windows-31j	0.0005	0.0005	0.0005
x-windows-874	0.0101	0.0085	0.0087
x-windows-949	0.0000	0.0000	0.0000