Character Encoding of HTML Pages

The character set or encoding of HTML pages only is identified by Tika’s AutoDetectReader. The table shows the percentage how character sets have been used to encode HTML pages crawled by the latest monthly crawls. The underlying data including page counts is provided in charsets.csv.

crawl	CC-MAIN-2024-30	CC-MAIN-2024-33	CC-MAIN-2024-38
charset
<other>	0.0001	0.0001	0.0000
<unknown>	1.9684	2.0394	1.5057
Big5	0.0779	0.0856	0.0823
Big5-HKSCS	0.0000	0.0000	0.0001
EUC-JP	0.1063	0.1060	0.1006
EUC-KR	0.1011	0.0984	0.0837
GB18030	0.0171	0.0170	0.0150
GB2312	0.2880	0.2748	0.2230
GBK	0.1383	0.1381	0.1110
IBM420	0.0052	0.0056	0.0040
IBM424	0.0021	0.0022	0.0014
IBM500	0.0007	0.0007	0.0007
IBM855	0.0000	0.0000	0.0000
IBM866	0.0002	0.0002	0.0003
ISO-2022-JP	0.0010	0.0012	0.0009
ISO-8859-1	2.2709	2.3973	2.7588
ISO-8859-13	0.0001	0.0001	0.0001
ISO-8859-15	0.0506	0.0528	0.0466
ISO-8859-16	0.0002	0.0002	0.0001
ISO-8859-2	0.1177	0.1278	0.1072
ISO-8859-3	0.0005	0.0006	0.0005
ISO-8859-4	0.0006	0.0007	0.0008
ISO-8859-5	0.0021	0.0022	0.0019
ISO-8859-6	0.0000	0.0000	0.0001
ISO-8859-7	0.0062	0.0067	0.0054
ISO-8859-8	0.0009	0.0010	0.0008
ISO-8859-9	0.0254	0.0275	0.0235
KOI8-R	0.0070	0.0071	0.0065
KOI8-U	0.0001	0.0001	0.0001
Shift_JIS	0.1747	0.1866	0.1602
TIS-620	0.0059	0.0060	0.0054
US-ASCII	0.0344	0.0378	0.0338
UTF-16	0.0054	0.0058	0.0047
UTF-16BE	0.0003	0.0003	0.0003
UTF-16LE	0.0016	0.0013	0.0011
UTF-32	0.0001	0.0000	0.0000
UTF-32LE	0.0002	0.0004	0.0002
UTF-8	93.6945	93.4387	93.9167
windows-1250	0.0789	0.0784	0.0696
windows-1251	0.5550	0.5710	0.4984
windows-1252	0.1767	0.1907	0.1552
windows-1253	0.0029	0.0028	0.0022
windows-1254	0.0117	0.0130	0.0111
windows-1255	0.0073	0.0078	0.0067
windows-1256	0.0448	0.0484	0.0380
windows-1257	0.0082	0.0079	0.0075
windows-1258	0.0000	0.0000	0.0000
windows-31j	0.0007	0.0006	0.0006
x-windows-874	0.0081	0.0092	0.0070
x-windows-949	0.0001	0.0001	0.0000