Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives
Latest crawl: CC-MAIN-2024-42
The language of a document is identified by Compact Language Detector 2 (CLD2). It is able to identify 160 different languages and up to 3 languages per document. The table lists the percentage covered by the primary language of a document (returned first by CLD2). So far, only HTML pages are passed to the language detector. The underlying data including page counts is provided in languages.csv.
crawl | CC-MAIN-2024-33 | CC-MAIN-2024-38 | CC-MAIN-2024-42 |
---|---|---|---|
language | |||
<unknown> | 3.2930 | 2.6706 | 3.2780 |
aar | 0.0001 | 0.0001 | 0.0001 |
abk | 0.0004 | 0.0005 | 0.0005 |
afr | 0.0104 | 0.0084 | 0.0103 |
aka | 0.0001 | 0.0001 | 0.0001 |
amh | 0.0029 | 0.0028 | 0.0030 |
ara | 0.7112 | 0.6568 | 0.6709 |
asm | 0.0023 | 0.0023 | 0.0024 |
aym | 0.0001 | 0.0001 | 0.0001 |
aze | 0.0740 | 0.0661 | 0.0687 |
bak | 0.0025 | 0.0028 | 0.0029 |
bel | 0.0159 | 0.0159 | 0.0162 |
ben | 0.1209 | 0.1075 | 0.1186 |
bih | 0.0007 | 0.0007 | 0.0006 |
bis | 0.0001 | 0.0001 | 0.0001 |
bod | 0.0013 | 0.0011 | 0.0015 |
bos | 0.0533 | 0.0484 | 0.0523 |
bre | 0.0032 | 0.0019 | 0.0034 |
bul | 0.3004 | 0.2941 | 0.2992 |
cat | 0.2045 | 0.2003 | 0.2085 |
ceb | 0.0020 | 0.0017 | 0.0020 |
ces | 1.0158 | 1.1019 | 1.0630 |
chr | 0.0000 | 0.0000 | 0.0000 |
cos | 0.0021 | 0.0019 | 0.0020 |
crs | 0.0002 | 0.0001 | 0.0001 |
cym | 0.0108 | 0.0100 | 0.0117 |
dan | 0.4557 | 0.4489 | 0.4640 |
deu | 5.0895 | 5.4471 | 5.3038 |
div | 0.0022 | 0.0021 | 0.0023 |
dzo | 0.0001 | 0.0001 | 0.0001 |
ell | 0.5803 | 0.5387 | 0.5728 |
eng | 43.1787 | 44.1210 | 43.4241 |
epo | 0.0096 | 0.0082 | 0.0084 |
est | 0.1302 | 0.1374 | 0.1326 |
eus | 0.0345 | 0.0333 | 0.0401 |
fao | 0.0045 | 0.0045 | 0.0048 |
fas | 0.7345 | 0.7190 | 0.7051 |
fij | 0.0001 | 0.0001 | 0.0001 |
fin | 0.3403 | 0.3538 | 0.3444 |
fra | 4.2439 | 4.4292 | 4.3960 |
fry | 0.0027 | 0.0020 | 0.0029 |
gla | 0.0015 | 0.0013 | 0.0014 |
gle | 0.0074 | 0.0070 | 0.0071 |
glg | 0.0359 | 0.0344 | 0.0365 |
glv | 0.0003 | 0.0004 | 0.0003 |
got | 0.0000 | 0.0000 | 0.0000 |
grn | 0.0008 | 0.0008 | 0.0007 |
guj | 0.0122 | 0.0113 | 0.0113 |
hat | 0.0019 | 0.0017 | 0.0019 |
hau | 0.0028 | 0.0023 | 0.0027 |
haw | 0.0007 | 0.0005 | 0.0007 |
heb | 0.2373 | 0.2379 | 0.2473 |
hin | 0.2018 | 0.1967 | 0.2089 |
hmn | 0.0006 | 0.0005 | 0.0006 |
hrv | 0.2224 | 0.2005 | 0.2149 |
hun | 0.5822 | 0.5725 | 0.5649 |
hye | 0.0366 | 0.0329 | 0.0340 |
ibo | 0.0006 | 0.0006 | 0.0007 |
iku | 0.0002 | 0.0002 | 0.0002 |
ile | 0.0005 | 0.0005 | 0.0004 |
ina | 0.0005 | 0.0005 | 0.0005 |
ind | 1.1139 | 1.0532 | 1.1235 |
ipk | 0.0001 | 0.0001 | 0.0001 |
isl | 0.0376 | 0.0390 | 0.0398 |
ita | 2.4921 | 2.5224 | 2.5282 |
jav | 0.0021 | 0.0020 | 0.0022 |
jpn | 4.8790 | 5.1119 | 5.0419 |
kal | 0.0008 | 0.0008 | 0.0009 |
kan | 0.0168 | 0.0145 | 0.0142 |
kas | 0.0000 | 0.0000 | 0.0000 |
kat | 0.0410 | 0.0391 | 0.0422 |
kaz | 0.0288 | 0.0253 | 0.0282 |
kha | 0.0003 | 0.0003 | 0.0004 |
khm | 0.0108 | 0.0103 | 0.0106 |
kin | 0.0022 | 0.0020 | 0.0023 |
kir | 0.0093 | 0.0086 | 0.0098 |
kor | 0.7224 | 0.6712 | 0.7388 |
kur | 0.0071 | 0.0069 | 0.0074 |
lao | 0.0035 | 0.0038 | 0.0037 |
lat | 0.1016 | 0.1144 | 0.1007 |
lav | 0.0912 | 0.0905 | 0.0942 |
lif | 0.0000 | 0.0000 | 0.0000 |
lin | 0.0004 | 0.0003 | 0.0003 |
lit | 0.1524 | 0.1618 | 0.1668 |
ltz | 0.0037 | 0.0037 | 0.0040 |
lug | 0.0002 | 0.0002 | 0.0002 |
mal | 0.0269 | 0.0244 | 0.0269 |
mar | 0.0278 | 0.0241 | 0.0258 |
mfe | 0.0002 | 0.0001 | 0.0001 |
mkd | 0.0422 | 0.0376 | 0.0383 |
mlg | 0.0034 | 0.0031 | 0.0036 |
mlt | 0.0047 | 0.0043 | 0.0049 |
mon | 0.0167 | 0.0149 | 0.0161 |
mri | 0.0013 | 0.0009 | 0.0012 |
msa | 0.0782 | 0.0739 | 0.0787 |
mya | 0.0139 | 0.0131 | 0.0140 |
nau | 0.0001 | 0.0000 | 0.0000 |
nep | 0.0533 | 0.0443 | 0.0493 |
nld | 1.7821 | 1.9149 | 1.8145 |
nno | 0.0126 | 0.0135 | 0.0130 |
nor | 0.2959 | 0.3091 | 0.3060 |
nso | 0.0000 | 0.0001 | 0.0001 |
nya | 0.0006 | 0.0005 | 0.0006 |
oci | 0.0033 | 0.0035 | 0.0034 |
ori | 0.0064 | 0.0053 | 0.0057 |
orm | 0.0006 | 0.0005 | 0.0005 |
pan | 0.0074 | 0.0068 | 0.0079 |
pol | 1.9342 | 1.8868 | 1.8065 |
por | 2.2179 | 2.2141 | 2.3146 |
pus | 0.0032 | 0.0032 | 0.0035 |
que | 0.0005 | 0.0006 | 0.0005 |
roh | 0.0009 | 0.0010 | 0.0010 |
ron | 0.6540 | 0.5556 | 0.5978 |
run | 0.0001 | 0.0001 | 0.0001 |
rus | 6.2242 | 6.1556 | 6.0444 |
sag | 0.0004 | 0.0004 | 0.0004 |
san | 0.0032 | 0.0030 | 0.0034 |
sco | 0.0007 | 0.0006 | 0.0007 |
sin | 0.0088 | 0.0085 | 0.0090 |
slk | 0.4056 | 0.4168 | 0.3705 |
slv | 0.1600 | 0.1556 | 0.1558 |
smo | 0.0007 | 0.0006 | 0.0007 |
sna | 0.0006 | 0.0006 | 0.0006 |
snd | 0.0020 | 0.0016 | 0.0017 |
som | 0.0050 | 0.0045 | 0.0050 |
sot | 0.0006 | 0.0005 | 0.0006 |
spa | 4.5819 | 4.4769 | 4.5387 |
sqi | 0.0508 | 0.0479 | 0.0497 |
srp | 0.2435 | 0.2245 | 0.2161 |
ssw | 0.0000 | 0.0000 | 0.0000 |
sun | 0.0010 | 0.0010 | 0.0010 |
sux | 0.0000 | 0.0000 | 0.0000 |
swa | 0.0100 | 0.0093 | 0.0096 |
swe | 0.6926 | 0.6859 | 0.6649 |
syr | 0.0003 | 0.0003 | 0.0003 |
tam | 0.0490 | 0.0439 | 0.0473 |
tat | 0.0072 | 0.0073 | 0.0087 |
tel | 0.0218 | 0.0181 | 0.0220 |
tgk | 0.0069 | 0.0060 | 0.0064 |
tgl | 0.0094 | 0.0090 | 0.0088 |
tha | 0.4309 | 0.4139 | 0.4071 |
tir | 0.0008 | 0.0008 | 0.0007 |
ton | 0.0001 | 0.0001 | 0.0001 |
tsn | 0.0001 | 0.0001 | 0.0001 |
tso | 0.0001 | 0.0000 | 0.0001 |
tuk | 0.0025 | 0.0019 | 0.0021 |
tur | 1.2744 | 1.2328 | 1.2422 |
uig | 0.0012 | 0.0010 | 0.0010 |
ukr | 0.6064 | 0.5629 | 0.6037 |
urd | 0.0327 | 0.0298 | 0.0306 |
uzb | 0.0248 | 0.0264 | 0.0293 |
ven | 0.0000 | 0.0000 | 0.0000 |
vie | 1.0514 | 1.0332 | 1.0454 |
vol | 0.0005 | 0.0005 | 0.0005 |
war | 0.0020 | 0.0019 | 0.0023 |
wol | 0.0002 | 0.0001 | 0.0001 |
xho | 0.0022 | 0.0018 | 0.0023 |
yid | 0.0018 | 0.0015 | 0.0017 |
yor | 0.0008 | 0.0008 | 0.0010 |
zha | 0.0001 | 0.0001 | 0.0001 |
zho | 5.1917 | 4.6266 | 4.8129 |
zul | 0.0028 | 0.0021 | 0.0025 |