Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives
Latest crawl: CC-MAIN-2023-06
The language of a document is identified by Compact Language Detector 2 (CLD2). It is able to identify 160 different languages and up to 3 languages per document. The table lists the percentage covered by the primary language of a document (returned first by CLD2). So far, only HTML pages are passed to the language detector. The underlying data including page counts is provided in languages.csv.
crawl | CC-MAIN-2022-40 | CC-MAIN-2022-49 | CC-MAIN-2023-06 |
---|---|---|---|
language | |||
<unknown> | 2.5673 | 2.7037 | 2.5188 |
aar | 0.0001 | 0.0002 | 0.0001 |
abk | 0.0008 | 0.0007 | 0.0006 |
afr | 0.0084 | 0.0079 | 0.0076 |
aka | 0.0001 | 0.0001 | 0.0001 |
amh | 0.0037 | 0.0031 | 0.0037 |
ara | 0.6658 | 0.6708 | 0.6756 |
asm | 0.0025 | 0.0025 | 0.0031 |
aym | 0.0001 | 0.0001 | 0.0001 |
aze | 0.0414 | 0.0402 | 0.0413 |
bak | 0.0013 | 0.0029 | 0.0027 |
bel | 0.0162 | 0.0156 | 0.0168 |
ben | 0.0930 | 0.0942 | 0.0963 |
bih | 0.0006 | 0.0007 | 0.0007 |
bis | 0.0001 | 0.0001 | 0.0001 |
bod | 0.0017 | 0.0015 | 0.0014 |
bos | 0.0473 | 0.0462 | 0.0477 |
bre | 0.0029 | 0.0027 | 0.0028 |
bul | 0.2900 | 0.2922 | 0.2970 |
cat | 0.2314 | 0.2369 | 0.2335 |
ceb | 0.0024 | 0.0022 | 0.0024 |
ces | 1.1069 | 1.0989 | 1.0994 |
chr | 0.0001 | 0.0001 | 0.0001 |
cos | 0.0026 | 0.0024 | 0.0025 |
crs | 0.0001 | 0.0001 | 0.0001 |
cym | 0.0095 | 0.0102 | 0.0100 |
dan | 0.4301 | 0.4473 | 0.4501 |
deu | 5.8811 | 5.8640 | 5.9010 |
div | 0.0021 | 0.0023 | 0.0023 |
dzo | 0.0002 | 0.0002 | 0.0002 |
ell | 0.5870 | 0.6068 | 0.6099 |
eng | 45.8786 | 46.3044 | 46.2733 |
epo | 0.0091 | 0.0091 | 0.0096 |
est | 0.1354 | 0.1368 | 0.1376 |
eus | 0.0323 | 0.0340 | 0.0332 |
fao | 0.0040 | 0.0042 | 0.0042 |
fas | 0.6582 | 0.6671 | 0.6589 |
fij | 0.0002 | 0.0001 | 0.0001 |
fin | 0.4080 | 0.4165 | 0.4156 |
fra | 4.7254 | 4.7768 | 4.7487 |
fry | 0.0032 | 0.0031 | 0.0029 |
gla | 0.0015 | 0.0016 | 0.0014 |
gle | 0.0063 | 0.0065 | 0.0066 |
glg | 0.0400 | 0.0408 | 0.0425 |
glv | 0.0005 | 0.0004 | 0.0004 |
got | 0.0000 | 0.0000 | 0.0000 |
grn | 0.0006 | 0.0006 | 0.0007 |
guj | 0.0126 | 0.0125 | 0.0128 |
hat | 0.0018 | 0.0018 | 0.0018 |
hau | 0.0028 | 0.0026 | 0.0031 |
haw | 0.0007 | 0.0007 | 0.0006 |
heb | 0.1152 | 0.1170 | 0.1150 |
hin | 0.1588 | 0.1592 | 0.1747 |
hmn | 0.0006 | 0.0007 | 0.0007 |
hrv | 0.1979 | 0.1990 | 0.2035 |
hun | 0.6093 | 0.6203 | 0.6018 |
hye | 0.0314 | 0.0315 | 0.0324 |
ibo | 0.0006 | 0.0006 | 0.0007 |
iku | 0.0002 | 0.0002 | 0.0002 |
ile | 0.0003 | 0.0003 | 0.0003 |
ina | 0.0007 | 0.0007 | 0.0007 |
ind | 0.7991 | 0.7992 | 0.8132 |
ipk | 0.0001 | 0.0001 | 0.0001 |
isl | 0.0397 | 0.0404 | 0.0418 |
ita | 2.5712 | 2.6835 | 2.6843 |
jav | 0.0023 | 0.0023 | 0.0025 |
jpn | 4.7884 | 4.4586 | 4.7648 |
kal | 0.0010 | 0.0010 | 0.0010 |
kan | 0.0122 | 0.0121 | 0.0135 |
kas | 0.0000 | 0.0000 | 0.0000 |
kat | 0.0352 | 0.0369 | 0.0358 |
kaz | 0.0211 | 0.0212 | 0.0205 |
kha | 0.0002 | 0.0002 | 0.0002 |
khm | 0.0108 | 0.0106 | 0.0131 |
kin | 0.0020 | 0.0019 | 0.0020 |
kir | 0.0049 | 0.0052 | 0.0053 |
kor | 0.6498 | 0.6526 | 0.6518 |
kur | 0.0055 | 0.0056 | 0.0054 |
lao | 0.0045 | 0.0041 | 0.0044 |
lat | 0.1232 | 0.1059 | 0.1158 |
lav | 0.0831 | 0.0832 | 0.0848 |
lif | 0.0000 | 0.0000 | 0.0000 |
lin | 0.0002 | 0.0002 | 0.0002 |
lit | 0.1617 | 0.1609 | 0.1626 |
ltz | 0.0037 | 0.0038 | 0.0040 |
lug | 0.0002 | 0.0002 | 0.0002 |
mal | 0.0222 | 0.0219 | 0.0227 |
mar | 0.0213 | 0.0212 | 0.0223 |
mfe | 0.0002 | 0.0002 | 0.0002 |
mkd | 0.0347 | 0.0364 | 0.0360 |
mlg | 0.0031 | 0.0027 | 0.0030 |
mlt | 0.0042 | 0.0043 | 0.0041 |
mon | 0.0216 | 0.0238 | 0.0169 |
mri | 0.0013 | 0.0013 | 0.0012 |
msa | 0.0826 | 0.0812 | 0.0812 |
mya | 0.0126 | 0.0118 | 0.0125 |
nau | 0.0001 | 0.0000 | 0.0000 |
nep | 0.0304 | 0.0306 | 0.0327 |
nld | 2.0585 | 2.1504 | 2.0693 |
nno | 0.0162 | 0.0159 | 0.0154 |
nor | 0.3421 | 0.3454 | 0.3401 |
nso | 0.0001 | 0.0001 | 0.0001 |
nya | 0.0006 | 0.0006 | 0.0006 |
oci | 0.0032 | 0.0033 | 0.0031 |
ori | 0.0044 | 0.0042 | 0.0049 |
orm | 0.0005 | 0.0004 | 0.0006 |
pan | 0.0061 | 0.0062 | 0.0063 |
pol | 1.6636 | 1.6881 | 1.6977 |
por | 1.1505 | 1.1535 | 1.1660 |
pus | 0.0028 | 0.0029 | 0.0030 |
que | 0.0007 | 0.0007 | 0.0007 |
roh | 0.0012 | 0.0012 | 0.0012 |
ron | 0.5637 | 0.5772 | 0.5727 |
run | 0.0001 | 0.0001 | 0.0001 |
rus | 5.9692 | 5.6647 | 5.7910 |
sag | 0.0002 | 0.0001 | 0.0002 |
san | 0.0029 | 0.0028 | 0.0032 |
sco | 0.0007 | 0.0006 | 0.0007 |
sin | 0.0085 | 0.0085 | 0.0078 |
slk | 0.3777 | 0.3890 | 0.3971 |
slv | 0.1582 | 0.1625 | 0.1600 |
smo | 0.0006 | 0.0005 | 0.0006 |
sna | 0.0007 | 0.0006 | 0.0006 |
snd | 0.0015 | 0.0014 | 0.0016 |
som | 0.0040 | 0.0039 | 0.0041 |
sot | 0.0007 | 0.0006 | 0.0006 |
spa | 4.4690 | 4.5435 | 4.5977 |
sqi | 0.0437 | 0.0450 | 0.0463 |
srp | 0.2205 | 0.2228 | 0.2236 |
ssw | 0.0000 | 0.0001 | 0.0000 |
sun | 0.0012 | 0.0011 | 0.0014 |
sux | 0.0000 | 0.0000 | 0.0000 |
swa | 0.0077 | 0.0075 | 0.0077 |
swe | 0.6969 | 0.6899 | 0.7016 |
syr | 0.0005 | 0.0003 | 0.0003 |
tam | 0.0446 | 0.0450 | 0.0463 |
tat | 0.0063 | 0.0078 | 0.0078 |
tel | 0.0183 | 0.0181 | 0.0189 |
tgk | 0.0047 | 0.0049 | 0.0050 |
tgl | 0.0088 | 0.0086 | 0.0073 |
tha | 0.4143 | 0.4027 | 0.4051 |
tir | 0.0009 | 0.0008 | 0.0010 |
ton | 0.0001 | 0.0001 | 0.0001 |
tsn | 0.0001 | 0.0001 | 0.0001 |
tso | 0.0000 | 0.0000 | 0.0000 |
tuk | 0.0027 | 0.0026 | 0.0027 |
tur | 0.8439 | 0.8040 | 0.8221 |
uig | 0.0015 | 0.0013 | 0.0014 |
ukr | 0.3304 | 0.3350 | 0.3358 |
urd | 0.0274 | 0.0277 | 0.0298 |
uzb | 0.0162 | 0.0150 | 0.0160 |
ven | 0.0000 | 0.0000 | 0.0000 |
vie | 1.0299 | 0.9937 | 1.0119 |
vol | 0.0006 | 0.0005 | 0.0005 |
war | 0.0027 | 0.0026 | 0.0024 |
wol | 0.0001 | 0.0001 | 0.0001 |
xho | 0.0015 | 0.0015 | 0.0011 |
yid | 0.0017 | 0.0015 | 0.0016 |
yor | 0.0010 | 0.0010 | 0.0010 |
zha | 0.0001 | 0.0001 | 0.0001 |
zho | 4.8747 | 4.5969 | 4.2807 |
zul | 0.0017 | 0.0017 | 0.0014 |