Character Encoding of HTML Pages

The character set or encoding of HTML pages only is identified by Tika’s AutoDetectReader. The table shows the percentage how character sets have been used to encode HTML pages crawled by the latest monthly crawls. The underlying data including page counts is provided in charsets.csv.

crawl	CC-MAIN-2024-33	CC-MAIN-2024-38	CC-MAIN-2024-42
charset
<other>	0.0001	0.0000	0.0001
<unknown>	2.0394	1.5057	2.0343
Big5	0.0856	0.0823	0.0861
Big5-HKSCS	0.0000	0.0001	0.0001
EUC-JP	0.1060	0.1006	0.1204
EUC-KR	0.0984	0.0837	0.0906
GB18030	0.0170	0.0150	0.0142
GB2312	0.2748	0.2230	0.2367
GBK	0.1381	0.1110	0.1094
IBM420	0.0056	0.0040	0.0047
IBM424	0.0022	0.0014	0.0019
IBM500	0.0007	0.0007	0.0006
IBM855	0.0000	0.0000	0.0000
IBM866	0.0002	0.0003	0.0003
ISO-2022-JP	0.0012	0.0009	0.0010
ISO-8859-1	2.3973	2.7588	2.6607
ISO-8859-13	0.0001	0.0001	0.0002
ISO-8859-15	0.0528	0.0466	0.0516
ISO-8859-16	0.0002	0.0001	0.0001
ISO-8859-2	0.1278	0.1072	0.1148
ISO-8859-3	0.0006	0.0005	0.0013
ISO-8859-4	0.0007	0.0008	0.0007
ISO-8859-5	0.0022	0.0019	0.0017
ISO-8859-6	0.0000	0.0001	0.0000
ISO-8859-7	0.0067	0.0054	0.0064
ISO-8859-8	0.0010	0.0008	0.0008
ISO-8859-9	0.0275	0.0235	0.0244
KOI8-R	0.0071	0.0065	0.0079
KOI8-U	0.0001	0.0001	0.0001
Shift_JIS	0.1866	0.1602	0.1888
TIS-620	0.0060	0.0054	0.0062
US-ASCII	0.0378	0.0338	0.0355
UTF-16	0.0058	0.0047	0.0058
UTF-16BE	0.0003	0.0003	0.0004
UTF-16LE	0.0013	0.0011	0.0014
UTF-32	0.0000	0.0000	0.0001
UTF-32LE	0.0004	0.0002	0.0002
UTF-8	93.4387	93.9167	93.3348
windows-1250	0.0784	0.0696	0.0785
windows-1251	0.5710	0.4984	0.5192
windows-1252	0.1907	0.1552	0.1798
windows-1253	0.0028	0.0022	0.0029
windows-1254	0.0130	0.0111	0.0121
windows-1255	0.0078	0.0067	0.0073
windows-1256	0.0484	0.0380	0.0403
windows-1257	0.0079	0.0075	0.0075
windows-1258	0.0000	0.0000	0.0000
windows-31j	0.0006	0.0006	0.0006
x-windows-874	0.0092	0.0070	0.0075
x-windows-949	0.0001	0.0000	0.0000