Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives
Latest crawl: CC-MAIN-2023-23
The language of a document is identified by Compact Language Detector 2 (CLD2). It is able to identify 160 different languages and up to 3 languages per document. The table lists the percentage covered by the primary language of a document (returned first by CLD2). So far, only HTML pages are passed to the language detector. The underlying data including page counts is provided in languages.csv.
crawl | CC-MAIN-2023-06 | CC-MAIN-2023-14 | CC-MAIN-2023-23 |
---|---|---|---|
language | |||
<unknown> | 2.5188 | 2.8514 | 2.8762 |
aar | 0.0001 | 0.0001 | 0.0002 |
abk | 0.0006 | 0.0005 | 0.0005 |
afr | 0.0076 | 0.0073 | 0.0070 |
aka | 0.0001 | 0.0001 | 0.0001 |
amh | 0.0037 | 0.0036 | 0.0037 |
ara | 0.6756 | 0.6790 | 0.6381 |
asm | 0.0031 | 0.0026 | 0.0025 |
aym | 0.0001 | 0.0001 | 0.0001 |
aze | 0.0413 | 0.0446 | 0.0440 |
bak | 0.0027 | 0.0026 | 0.0027 |
bel | 0.0168 | 0.0166 | 0.0151 |
ben | 0.0963 | 0.1036 | 0.0965 |
bih | 0.0007 | 0.0008 | 0.0007 |
bis | 0.0001 | 0.0001 | 0.0001 |
bod | 0.0014 | 0.0014 | 0.0013 |
bos | 0.0477 | 0.0462 | 0.0461 |
bre | 0.0028 | 0.0025 | 0.0028 |
bul | 0.2970 | 0.2961 | 0.3062 |
cat | 0.2335 | 0.2363 | 0.2433 |
ceb | 0.0024 | 0.0023 | 0.0016 |
ces | 1.0994 | 1.1069 | 1.1482 |
chr | 0.0001 | 0.0000 | 0.0001 |
cos | 0.0025 | 0.0025 | 0.0024 |
crs | 0.0001 | 0.0001 | 0.0001 |
cym | 0.0100 | 0.0101 | 0.0094 |
dan | 0.4501 | 0.4384 | 0.4287 |
deu | 5.9010 | 5.7651 | 6.0319 |
div | 0.0023 | 0.0024 | 0.0023 |
dzo | 0.0002 | 0.0001 | 0.0001 |
ell | 0.6099 | 0.6088 | 0.5938 |
eng | 46.2733 | 46.1750 | 45.2170 |
epo | 0.0096 | 0.0096 | 0.0090 |
est | 0.1376 | 0.1387 | 0.1410 |
eus | 0.0332 | 0.0350 | 0.0346 |
fao | 0.0042 | 0.0042 | 0.0041 |
fas | 0.6589 | 0.6002 | 0.7105 |
fij | 0.0001 | 0.0001 | 0.0001 |
fin | 0.4156 | 0.3965 | 0.4007 |
fra | 4.7487 | 4.6770 | 4.7691 |
fry | 0.0029 | 0.0028 | 0.0029 |
gla | 0.0014 | 0.0016 | 0.0015 |
gle | 0.0066 | 0.0070 | 0.0065 |
glg | 0.0425 | 0.0418 | 0.0417 |
glv | 0.0004 | 0.0003 | 0.0004 |
got | 0.0000 | 0.0000 | 0.0000 |
grn | 0.0007 | 0.0006 | 0.0006 |
guj | 0.0128 | 0.0142 | 0.0142 |
hat | 0.0018 | 0.0020 | 0.0019 |
hau | 0.0031 | 0.0036 | 0.0035 |
haw | 0.0006 | 0.0008 | 0.0006 |
heb | 0.1150 | 0.1140 | 0.1110 |
hin | 0.1747 | 0.1790 | 0.1770 |
hmn | 0.0007 | 0.0008 | 0.0007 |
hrv | 0.2035 | 0.1882 | 0.1857 |
hun | 0.6018 | 0.6120 | 0.6161 |
hye | 0.0324 | 0.0337 | 0.0307 |
ibo | 0.0007 | 0.0008 | 0.0006 |
iku | 0.0002 | 0.0002 | 0.0002 |
ile | 0.0003 | 0.0003 | 0.0003 |
ina | 0.0007 | 0.0006 | 0.0005 |
ind | 0.8132 | 0.7898 | 0.7676 |
ipk | 0.0001 | 0.0001 | 0.0001 |
isl | 0.0418 | 0.0400 | 0.0412 |
ita | 2.6843 | 2.7261 | 2.7524 |
jav | 0.0025 | 0.0046 | 0.0023 |
jpn | 4.7648 | 4.4368 | 5.0335 |
kal | 0.0010 | 0.0007 | 0.0009 |
kan | 0.0135 | 0.0134 | 0.0132 |
kas | 0.0000 | 0.0000 | 0.0000 |
kat | 0.0358 | 0.0383 | 0.0369 |
kaz | 0.0205 | 0.0224 | 0.0214 |
kha | 0.0002 | 0.0003 | 0.0003 |
khm | 0.0131 | 0.0245 | 0.0110 |
kin | 0.0020 | 0.0021 | 0.0018 |
kir | 0.0053 | 0.0051 | 0.0051 |
kor | 0.6518 | 0.6576 | 0.6760 |
kur | 0.0054 | 0.0062 | 0.0059 |
lao | 0.0044 | 0.0043 | 0.0039 |
lat | 0.1158 | 0.1099 | 0.1179 |
lav | 0.0848 | 0.0855 | 0.0864 |
lif | 0.0000 | 0.0000 | 0.0000 |
lin | 0.0002 | 0.0002 | 0.0002 |
lit | 0.1626 | 0.1668 | 0.1683 |
ltz | 0.0040 | 0.0041 | 0.0043 |
lug | 0.0002 | 0.0002 | 0.0002 |
mal | 0.0227 | 0.0231 | 0.0229 |
mar | 0.0223 | 0.0249 | 0.0237 |
mfe | 0.0002 | 0.0001 | 0.0001 |
mkd | 0.0360 | 0.0356 | 0.0341 |
mlg | 0.0030 | 0.0026 | 0.0029 |
mlt | 0.0041 | 0.0042 | 0.0043 |
mon | 0.0169 | 0.0165 | 0.0161 |
mri | 0.0012 | 0.0013 | 0.0011 |
msa | 0.0812 | 0.0774 | 0.0785 |
mya | 0.0125 | 0.0127 | 0.0116 |
nau | 0.0000 | 0.0000 | 0.0000 |
nep | 0.0327 | 0.0327 | 0.0327 |
nld | 2.0693 | 2.1208 | 2.1337 |
nno | 0.0154 | 0.0165 | 0.0155 |
nor | 0.3401 | 0.3462 | 0.3404 |
nso | 0.0001 | 0.0000 | 0.0001 |
nya | 0.0006 | 0.0008 | 0.0006 |
oci | 0.0031 | 0.0033 | 0.0033 |
ori | 0.0049 | 0.0053 | 0.0052 |
orm | 0.0006 | 0.0005 | 0.0005 |
pan | 0.0063 | 0.0069 | 0.0062 |
pol | 1.6977 | 1.6429 | 1.6840 |
por | 1.1660 | 1.1307 | 1.1399 |
pus | 0.0030 | 0.0033 | 0.0032 |
que | 0.0007 | 0.0007 | 0.0007 |
roh | 0.0012 | 0.0013 | 0.0014 |
ron | 0.5727 | 0.5272 | 0.4999 |
run | 0.0001 | 0.0001 | 0.0001 |
rus | 5.7910 | 5.8227 | 5.9898 |
sag | 0.0002 | 0.0001 | 0.0001 |
san | 0.0032 | 0.0032 | 0.0032 |
sco | 0.0007 | 0.0006 | 0.0007 |
sin | 0.0078 | 0.0081 | 0.0080 |
slk | 0.3971 | 0.3984 | 0.4032 |
slv | 0.1600 | 0.1627 | 0.1656 |
smo | 0.0006 | 0.0006 | 0.0005 |
sna | 0.0006 | 0.0007 | 0.0006 |
snd | 0.0016 | 0.0017 | 0.0014 |
som | 0.0041 | 0.0042 | 0.0039 |
sot | 0.0006 | 0.0007 | 0.0006 |
spa | 4.5977 | 4.5480 | 4.6013 |
sqi | 0.0463 | 0.0475 | 0.0435 |
srp | 0.2236 | 0.2160 | 0.2182 |
ssw | 0.0000 | 0.0000 | 0.0000 |
sun | 0.0014 | 0.0011 | 0.0011 |
sux | 0.0000 | 0.0000 | 0.0000 |
swa | 0.0077 | 0.0080 | 0.0084 |
swe | 0.7016 | 0.7001 | 0.7215 |
syr | 0.0003 | 0.0003 | 0.0003 |
tam | 0.0463 | 0.0452 | 0.0460 |
tat | 0.0078 | 0.0083 | 0.0064 |
tel | 0.0189 | 0.0194 | 0.0184 |
tgk | 0.0050 | 0.0051 | 0.0054 |
tgl | 0.0073 | 0.0072 | 0.0072 |
tha | 0.4051 | 0.4402 | 0.4061 |
tir | 0.0010 | 0.0008 | 0.0009 |
ton | 0.0001 | 0.0001 | 0.0001 |
tsn | 0.0001 | 0.0001 | 0.0001 |
tso | 0.0000 | 0.0000 | 0.0000 |
tuk | 0.0027 | 0.0022 | 0.0020 |
tur | 0.8221 | 0.7897 | 0.7667 |
uig | 0.0014 | 0.0015 | 0.0013 |
ukr | 0.3358 | 0.3633 | 0.3813 |
urd | 0.0298 | 0.0291 | 0.0273 |
uzb | 0.0160 | 0.0173 | 0.0168 |
ven | 0.0000 | 0.0000 | 0.0000 |
vie | 1.0119 | 1.0555 | 1.0275 |
vol | 0.0005 | 0.0005 | 0.0006 |
war | 0.0024 | 0.0026 | 0.0019 |
wol | 0.0001 | 0.0001 | 0.0001 |
xho | 0.0011 | 0.0011 | 0.0019 |
yid | 0.0016 | 0.0018 | 0.0016 |
yor | 0.0010 | 0.0011 | 0.0010 |
zha | 0.0001 | 0.0001 | 0.0001 |
zho | 4.2807 | 4.6316 | 4.3037 |
zul | 0.0014 | 0.0016 | 0.0020 |