Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives
Latest crawl: CC-MAIN-2024-51
The language of a document is identified by Compact Language Detector 2 (CLD2). It is able to identify 160 different languages and up to 3 languages per document. The table lists the percentage covered by the primary language of a document (returned first by CLD2). So far, only HTML pages are passed to the language detector. The underlying data including page counts is provided in languages.csv.
crawl | CC-MAIN-2024-42 | CC-MAIN-2024-46 | CC-MAIN-2024-51 |
---|---|---|---|
language | |||
<unknown> | 3.2780 | 3.0185 | 3.0651 |
aar | 0.0001 | 0.0001 | 0.0001 |
abk | 0.0005 | 0.0005 | 0.0005 |
afr | 0.0103 | 0.0097 | 0.0095 |
aka | 0.0001 | 0.0001 | 0.0001 |
amh | 0.0030 | 0.0028 | 0.0025 |
ara | 0.6709 | 0.6825 | 0.6837 |
asm | 0.0024 | 0.0026 | 0.0025 |
aym | 0.0001 | 0.0001 | 0.0001 |
aze | 0.0687 | 0.0632 | 0.0615 |
bak | 0.0029 | 0.0028 | 0.0025 |
bel | 0.0162 | 0.0159 | 0.0148 |
ben | 0.1186 | 0.1157 | 0.1134 |
bih | 0.0006 | 0.0005 | 0.0005 |
bis | 0.0001 | 0.0001 | 0.0001 |
bod | 0.0015 | 0.0014 | 0.0015 |
bos | 0.0523 | 0.0517 | 0.0525 |
bre | 0.0034 | 0.0022 | 0.0034 |
bul | 0.2992 | 0.2913 | 0.2983 |
cat | 0.2085 | 0.1998 | 0.1990 |
ceb | 0.0020 | 0.0019 | 0.0018 |
ces | 1.0630 | 1.0182 | 0.9955 |
chr | 0.0000 | 0.0000 | 0.0000 |
cos | 0.0020 | 0.0021 | 0.0019 |
crs | 0.0001 | 0.0001 | 0.0001 |
cym | 0.0117 | 0.0121 | 0.0110 |
dan | 0.4640 | 0.4702 | 0.4437 |
deu | 5.3038 | 5.3362 | 5.3963 |
div | 0.0023 | 0.0023 | 0.0025 |
dzo | 0.0001 | 0.0002 | 0.0002 |
ell | 0.5728 | 0.5791 | 0.5779 |
eng | 43.4241 | 42.9924 | 42.7983 |
epo | 0.0084 | 0.0087 | 0.0071 |
est | 0.1326 | 0.1332 | 0.1325 |
eus | 0.0401 | 0.0360 | 0.0367 |
fao | 0.0048 | 0.0045 | 0.0042 |
fas | 0.7051 | 0.7387 | 0.7441 |
fij | 0.0001 | 0.0001 | 0.0001 |
fin | 0.3444 | 0.3459 | 0.3510 |
fra | 4.3960 | 4.3541 | 4.4110 |
fry | 0.0029 | 0.0020 | 0.0027 |
gla | 0.0014 | 0.0013 | 0.0013 |
gle | 0.0071 | 0.0065 | 0.0069 |
glg | 0.0365 | 0.0325 | 0.0323 |
glv | 0.0003 | 0.0004 | 0.0003 |
got | 0.0000 | 0.0000 | 0.0000 |
grn | 0.0007 | 0.0008 | 0.0008 |
guj | 0.0113 | 0.0123 | 0.0117 |
hat | 0.0019 | 0.0019 | 0.0019 |
hau | 0.0027 | 0.0026 | 0.0027 |
haw | 0.0007 | 0.0007 | 0.0006 |
heb | 0.2473 | 0.2518 | 0.2501 |
hin | 0.2089 | 0.2066 | 0.1995 |
hmn | 0.0006 | 0.0006 | 0.0006 |
hrv | 0.2149 | 0.2248 | 0.2210 |
hun | 0.5649 | 0.5680 | 0.5171 |
hye | 0.0340 | 0.0340 | 0.0335 |
ibo | 0.0007 | 0.0006 | 0.0007 |
iku | 0.0002 | 0.0002 | 0.0002 |
ile | 0.0004 | 0.0004 | 0.0003 |
ina | 0.0005 | 0.0006 | 0.0005 |
ind | 1.1235 | 1.1186 | 1.1051 |
ipk | 0.0001 | 0.0001 | 0.0001 |
isl | 0.0398 | 0.0400 | 0.0408 |
ita | 2.5282 | 2.4952 | 2.4139 |
jav | 0.0022 | 0.0021 | 0.0023 |
jpn | 5.0419 | 5.1463 | 5.2819 |
kal | 0.0009 | 0.0009 | 0.0008 |
kan | 0.0142 | 0.0142 | 0.0133 |
kas | 0.0000 | 0.0000 | 0.0000 |
kat | 0.0422 | 0.0412 | 0.0399 |
kaz | 0.0282 | 0.0272 | 0.0284 |
kha | 0.0004 | 0.0004 | 0.0004 |
khm | 0.0106 | 0.0105 | 0.0102 |
kin | 0.0023 | 0.0021 | 0.0022 |
kir | 0.0098 | 0.0091 | 0.0103 |
kor | 0.7388 | 0.7228 | 0.7580 |
kur | 0.0074 | 0.0071 | 0.0068 |
lao | 0.0037 | 0.0035 | 0.0032 |
lat | 0.1007 | 0.1270 | 0.1039 |
lav | 0.0942 | 0.0902 | 0.0898 |
lif | 0.0000 | 0.0000 | 0.0000 |
lin | 0.0003 | 0.0003 | 0.0002 |
lit | 0.1668 | 0.1599 | 0.1618 |
ltz | 0.0040 | 0.0039 | 0.0038 |
lug | 0.0002 | 0.0002 | 0.0001 |
mal | 0.0269 | 0.0269 | 0.0254 |
mar | 0.0258 | 0.0274 | 0.0271 |
mfe | 0.0001 | 0.0001 | 0.0002 |
mkd | 0.0383 | 0.0383 | 0.0370 |
mlg | 0.0036 | 0.0029 | 0.0032 |
mlt | 0.0049 | 0.0044 | 0.0034 |
mon | 0.0161 | 0.0152 | 0.0155 |
mri | 0.0012 | 0.0011 | 0.0012 |
msa | 0.0787 | 0.0770 | 0.0771 |
mya | 0.0140 | 0.0130 | 0.0128 |
nau | 0.0000 | 0.0000 | 0.0000 |
nep | 0.0493 | 0.0522 | 0.0521 |
nld | 1.8145 | 1.8271 | 1.8093 |
nno | 0.0130 | 0.0138 | 0.0164 |
nor | 0.3060 | 0.3004 | 0.3023 |
nso | 0.0001 | 0.0001 | 0.0001 |
nya | 0.0006 | 0.0005 | 0.0005 |
oci | 0.0034 | 0.0033 | 0.0039 |
ori | 0.0057 | 0.0058 | 0.0061 |
orm | 0.0005 | 0.0005 | 0.0004 |
pan | 0.0079 | 0.0080 | 0.0075 |
pol | 1.8065 | 1.8484 | 1.9072 |
por | 2.3146 | 2.3292 | 2.3361 |
pus | 0.0035 | 0.0032 | 0.0033 |
que | 0.0005 | 0.0006 | 0.0005 |
roh | 0.0010 | 0.0011 | 0.0011 |
ron | 0.5978 | 0.6165 | 0.6095 |
run | 0.0001 | 0.0001 | 0.0001 |
rus | 6.0444 | 6.2093 | 6.1830 |
sag | 0.0004 | 0.0004 | 0.0003 |
san | 0.0034 | 0.0033 | 0.0031 |
sco | 0.0007 | 0.0007 | 0.0006 |
sin | 0.0090 | 0.0088 | 0.0093 |
slk | 0.3705 | 0.3708 | 0.3594 |
slv | 0.1558 | 0.1625 | 0.1577 |
smo | 0.0007 | 0.0006 | 0.0006 |
sna | 0.0006 | 0.0006 | 0.0007 |
snd | 0.0017 | 0.0015 | 0.0014 |
som | 0.0050 | 0.0050 | 0.0047 |
sot | 0.0006 | 0.0006 | 0.0006 |
spa | 4.5387 | 4.6154 | 4.5824 |
sqi | 0.0497 | 0.0496 | 0.0457 |
srp | 0.2161 | 0.2211 | 0.2203 |
ssw | 0.0000 | 0.0000 | 0.0000 |
sun | 0.0010 | 0.0010 | 0.0010 |
sux | 0.0000 | 0.0000 | 0.0000 |
swa | 0.0096 | 0.0091 | 0.0083 |
swe | 0.6649 | 0.6578 | 0.6438 |
syr | 0.0003 | 0.0004 | 0.0003 |
tam | 0.0473 | 0.0449 | 0.0437 |
tat | 0.0087 | 0.0076 | 0.0084 |
tel | 0.0220 | 0.0199 | 0.0201 |
tgk | 0.0064 | 0.0067 | 0.0070 |
tgl | 0.0088 | 0.0088 | 0.0085 |
tha | 0.4071 | 0.4228 | 0.4200 |
tir | 0.0007 | 0.0007 | 0.0007 |
ton | 0.0001 | 0.0001 | 0.0001 |
tsn | 0.0001 | 0.0001 | 0.0001 |
tso | 0.0001 | 0.0001 | 0.0001 |
tuk | 0.0021 | 0.0021 | 0.0021 |
tur | 1.2422 | 1.2479 | 1.2537 |
uig | 0.0010 | 0.0011 | 0.0012 |
ukr | 0.6037 | 0.6097 | 0.6181 |
urd | 0.0306 | 0.0300 | 0.0292 |
uzb | 0.0293 | 0.0228 | 0.0221 |
ven | 0.0000 | 0.0000 | 0.0000 |
vie | 1.0454 | 1.0347 | 1.0133 |
vol | 0.0005 | 0.0005 | 0.0004 |
war | 0.0023 | 0.0020 | 0.0021 |
wol | 0.0001 | 0.0001 | 0.0001 |
xho | 0.0023 | 0.0021 | 0.0008 |
yid | 0.0017 | 0.0015 | 0.0015 |
yor | 0.0010 | 0.0009 | 0.0009 |
zha | 0.0001 | 0.0001 | 0.0001 |
zho | 4.8129 | 5.1252 | 5.2720 |
zul | 0.0025 | 0.0025 | 0.0013 |