Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives
Latest crawl: CC-MAIN-2024-26
The language of a document is identified by Compact Language Detector 2 (CLD2). It is able to identify 160 different languages and up to 3 languages per document. The table lists the percentage covered by the primary language of a document (returned first by CLD2). So far, only HTML pages are passed to the language detector. The underlying data including page counts is provided in languages.csv.
crawl | CC-MAIN-2024-18 | CC-MAIN-2024-22 | CC-MAIN-2024-26 |
---|---|---|---|
language | |||
<unknown> | 2.8748 | 2.8592 | 3.1360 |
aar | 0.0001 | 0.0001 | 0.0001 |
abk | 0.0005 | 0.0005 | 0.0005 |
afr | 0.0090 | 0.0092 | 0.0085 |
aka | 0.0001 | 0.0001 | 0.0001 |
amh | 0.0032 | 0.0033 | 0.0032 |
ara | 0.6178 | 0.6353 | 0.6568 |
asm | 0.0022 | 0.0021 | 0.0021 |
aym | 0.0001 | 0.0001 | 0.0001 |
aze | 0.0617 | 0.0591 | 0.0623 |
bak | 0.0028 | 0.0027 | 0.0027 |
bel | 0.0185 | 0.0197 | 0.0151 |
ben | 0.1063 | 0.1078 | 0.1153 |
bih | 0.0007 | 0.0006 | 0.0007 |
bis | 0.0001 | 0.0001 | 0.0001 |
bod | 0.0017 | 0.0016 | 0.0013 |
bos | 0.0510 | 0.0506 | 0.0506 |
bre | 0.0027 | 0.0020 | 0.0030 |
bul | 0.2927 | 0.2883 | 0.2901 |
cat | 0.2036 | 0.2046 | 0.2011 |
ceb | 0.0021 | 0.0021 | 0.0019 |
ces | 1.0126 | 1.1223 | 1.0904 |
chr | 0.0000 | 0.0000 | 0.0000 |
cos | 0.0021 | 0.0020 | 0.0020 |
crs | 0.0001 | 0.0001 | 0.0001 |
cym | 0.0107 | 0.0113 | 0.0109 |
dan | 0.4291 | 0.4692 | 0.4529 |
deu | 5.2631 | 5.5710 | 5.4701 |
div | 0.0024 | 0.0020 | 0.0022 |
dzo | 0.0001 | 0.0002 | 0.0002 |
ell | 0.5579 | 0.5513 | 0.5475 |
eng | 45.5105 | 45.9693 | 45.0546 |
epo | 0.0092 | 0.0084 | 0.0084 |
est | 0.1298 | 0.1407 | 0.1381 |
eus | 0.0331 | 0.0349 | 0.0350 |
fao | 0.0040 | 0.0042 | 0.0040 |
fas | 0.6875 | 0.6807 | 0.6430 |
fij | 0.0001 | 0.0001 | 0.0001 |
fin | 0.3621 | 0.3513 | 0.3498 |
fra | 4.3078 | 4.2157 | 4.2175 |
fry | 0.0031 | 0.0022 | 0.0029 |
gla | 0.0013 | 0.0014 | 0.0013 |
gle | 0.0068 | 0.0066 | 0.0067 |
glg | 0.0361 | 0.0383 | 0.0392 |
glv | 0.0004 | 0.0003 | 0.0004 |
got | 0.0000 | 0.0000 | 0.0000 |
grn | 0.0007 | 0.0007 | 0.0007 |
guj | 0.0120 | 0.0117 | 0.0119 |
hat | 0.0018 | 0.0018 | 0.0019 |
hau | 0.0025 | 0.0025 | 0.0030 |
haw | 0.0005 | 0.0006 | 0.0006 |
heb | 0.2167 | 0.2254 | 0.2241 |
hin | 0.1859 | 0.1856 | 0.1884 |
hmn | 0.0005 | 0.0005 | 0.0007 |
hrv | 0.2050 | 0.1958 | 0.1961 |
hun | 0.5826 | 0.5784 | 0.5585 |
hye | 0.0325 | 0.0319 | 0.0329 |
ibo | 0.0005 | 0.0006 | 0.0006 |
iku | 0.0002 | 0.0002 | 0.0002 |
ile | 0.0003 | 0.0003 | 0.0004 |
ina | 0.0005 | 0.0005 | 0.0005 |
ind | 0.9190 | 0.8905 | 0.9675 |
ipk | 0.0000 | 0.0000 | 0.0001 |
isl | 0.0389 | 0.0371 | 0.0385 |
ita | 2.6120 | 2.4205 | 2.6379 |
jav | 0.0021 | 0.0020 | 0.0021 |
jpn | 5.1574 | 4.9833 | 4.7772 |
kal | 0.0008 | 0.0009 | 0.0008 |
kan | 0.0131 | 0.0145 | 0.0152 |
kas | 0.0000 | 0.0000 | 0.0000 |
kat | 0.0375 | 0.0370 | 0.0391 |
kaz | 0.0240 | 0.0250 | 0.0267 |
kha | 0.0002 | 0.0003 | 0.0004 |
khm | 0.0110 | 0.0107 | 0.0110 |
kin | 0.0021 | 0.0023 | 0.0022 |
kir | 0.0085 | 0.0093 | 0.0082 |
kor | 0.6913 | 0.7008 | 0.7992 |
kur | 0.0061 | 0.0062 | 0.0058 |
lao | 0.0038 | 0.0038 | 0.0034 |
lat | 0.1141 | 0.1169 | 0.1162 |
lav | 0.0862 | 0.0846 | 0.0880 |
lif | 0.0000 | 0.0000 | 0.0000 |
lin | 0.0002 | 0.0002 | 0.0003 |
lit | 0.1561 | 0.1527 | 0.1577 |
ltz | 0.0038 | 0.0037 | 0.0037 |
lug | 0.0002 | 0.0002 | 0.0002 |
mal | 0.0221 | 0.0226 | 0.0231 |
mar | 0.0247 | 0.0276 | 0.0263 |
mfe | 0.0001 | 0.0002 | 0.0002 |
mkd | 0.0380 | 0.0373 | 0.0394 |
mlg | 0.0031 | 0.0035 | 0.0035 |
mlt | 0.0044 | 0.0047 | 0.0043 |
mon | 0.0140 | 0.0144 | 0.0152 |
mri | 0.0011 | 0.0009 | 0.0011 |
msa | 0.0746 | 0.0740 | 0.0772 |
mya | 0.0126 | 0.0127 | 0.0133 |
nau | 0.0000 | 0.0000 | 0.0001 |
nep | 0.0445 | 0.0437 | 0.0490 |
nld | 1.9101 | 1.8350 | 1.8288 |
nno | 0.0142 | 0.0147 | 0.0131 |
nor | 0.3159 | 0.3066 | 0.3024 |
nso | 0.0001 | 0.0001 | 0.0001 |
nya | 0.0005 | 0.0005 | 0.0006 |
oci | 0.0033 | 0.0036 | 0.0031 |
ori | 0.0051 | 0.0054 | 0.0059 |
orm | 0.0006 | 0.0006 | 0.0006 |
pan | 0.0059 | 0.0058 | 0.0060 |
pol | 1.7582 | 1.7290 | 1.6771 |
por | 1.9505 | 2.0407 | 2.0232 |
pus | 0.0034 | 0.0037 | 0.0036 |
que | 0.0006 | 0.0005 | 0.0005 |
roh | 0.0010 | 0.0011 | 0.0011 |
ron | 0.5460 | 0.5208 | 0.5164 |
run | 0.0001 | 0.0001 | 0.0001 |
rus | 5.9477 | 5.9154 | 5.6876 |
sag | 0.0005 | 0.0005 | 0.0003 |
san | 0.0035 | 0.0035 | 0.0028 |
sco | 0.0006 | 0.0006 | 0.0010 |
sin | 0.0083 | 0.0097 | 0.0085 |
slk | 0.3673 | 0.3676 | 0.3705 |
slv | 0.1575 | 0.1469 | 0.1525 |
smo | 0.0006 | 0.0006 | 0.0006 |
sna | 0.0006 | 0.0006 | 0.0006 |
snd | 0.0016 | 0.0015 | 0.0017 |
som | 0.0047 | 0.0046 | 0.0045 |
sot | 0.0006 | 0.0006 | 0.0006 |
spa | 4.5941 | 4.6177 | 4.5025 |
sqi | 0.0470 | 0.0479 | 0.0483 |
srp | 0.2206 | 0.2178 | 0.2184 |
ssw | 0.0000 | 0.0000 | 0.0001 |
sun | 0.0009 | 0.0010 | 0.0010 |
sux | 0.0000 | 0.0000 | 0.0000 |
swa | 0.0080 | 0.0090 | 0.0095 |
swe | 0.6498 | 0.6337 | 0.6279 |
syr | 0.0003 | 0.0003 | 0.0003 |
tam | 0.0424 | 0.0410 | 0.0434 |
tat | 0.0068 | 0.0072 | 0.0071 |
tel | 0.0193 | 0.0208 | 0.0210 |
tgk | 0.0064 | 0.0063 | 0.0064 |
tgl | 0.0088 | 0.0144 | 0.0122 |
tha | 0.4145 | 0.3988 | 0.4189 |
tir | 0.0008 | 0.0008 | 0.0008 |
ton | 0.0001 | 0.0001 | 0.0001 |
tsn | 0.0001 | 0.0001 | 0.0001 |
tso | 0.0000 | 0.0000 | 0.0001 |
tuk | 0.0018 | 0.0019 | 0.0019 |
tur | 1.0623 | 1.1142 | 1.1946 |
uig | 0.0013 | 0.0011 | 0.0012 |
ukr | 0.4501 | 0.4461 | 0.5364 |
urd | 0.0312 | 0.0307 | 0.0298 |
uzb | 0.0209 | 0.0203 | 0.0208 |
ven | 0.0000 | 0.0000 | 0.0000 |
vie | 0.9784 | 0.9609 | 0.9859 |
vol | 0.0005 | 0.0006 | 0.0005 |
war | 0.0022 | 0.0021 | 0.0021 |
wol | 0.0001 | 0.0001 | 0.0001 |
xho | 0.0017 | 0.0019 | 0.0020 |
yid | 0.0017 | 0.0018 | 0.0018 |
yor | 0.0008 | 0.0008 | 0.0008 |
zha | 0.0001 | 0.0001 | 0.0001 |
zho | 4.4238 | 4.0651 | 4.8848 |
zul | 0.0020 | 0.0022 | 0.0021 |