Statistics of Common Crawl Monthly Archives

Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives
Latest crawl: CC-MAIN-2025-43

View the Project on GitHub

Distribution of Languages

The language of a document is identified by Compact Language Detector 2 (CLD2). It is able to identify 160 different languages and up to 3 languages per document. The table lists the percentage covered by the primary language of a document (returned first by CLD2). So far, only HTML pages are passed to the language detector. The underlying data including page counts is provided in languages.csv.

crawl CC-MAIN-2025-33 CC-MAIN-2025-38 CC-MAIN-2025-43
language
<unknown> 2.7925 2.7649 2.5819
aar 0.0001 0.0001 0.0001
abk 0.0006 0.0006 0.0006
afr 0.0086 0.0082 0.0082
aka 0.0002 0.0001 0.0001
amh 0.0032 0.0033 0.0034
ara 0.6383 0.6549 0.6604
asm 0.0025 0.0025 0.0025
aym 0.0001 0.0001 0.0001
aze 0.0576 0.0624 0.0542
bak 0.0015 0.0014 0.0008
bel 0.0150 0.0153 0.0133
ben 0.1040 0.1093 0.1112
bih 0.0005 0.0005 0.0006
bis 0.0001 0.0001 0.0001
bod 0.0013 0.0009 0.0011
bos 0.0530 0.0557 0.0494
bre 0.0033 0.0016 0.0031
bul 0.2584 0.2840 0.2577
cat 0.1732 0.1741 0.1775
ceb 0.0019 0.0021 0.0018
ces 1.0202 0.9881 1.0675
chr 0.0000 0.0000 0.0000
cos 0.0022 0.0022 0.0021
crs 0.0001 0.0001 0.0001
cym 0.0100 0.0094 0.0100
dan 0.4822 0.4432 0.5058
deu 5.8191 6.0060 5.6512
div 0.0024 0.0025 0.0021
dzo 0.0001 0.0001 0.0001
ell 0.5255 0.4898 0.4899
eng 44.2668 44.3285 44.8292
epo 0.0078 0.0073 0.0068
est 0.1182 0.1170 0.1156
eus 0.0308 0.0306 0.0303
fao 0.0047 0.0045 0.0047
fas 0.7886 0.7814 0.7378
fij 0.0001 0.0001 0.0001
fin 0.3422 0.3184 0.3372
fra 4.4559 4.4605 4.4414
fry 0.0017 0.0032 0.0014
gla 0.0015 0.0014 0.0014
gle 0.0070 0.0069 0.0057
glg 0.0272 0.0279 0.0260
glv 0.0004 0.0004 0.0004
got 0.0000 0.0000 0.0000
grn 0.0008 0.0008 0.0008
guj 0.0106 0.0110 0.0116
hat 0.0025 0.0017 0.0023
hau 0.0024 0.0027 0.0025
haw 0.0007 0.0008 0.0007
heb 0.2450 0.2285 0.2449
hin 0.1927 0.1930 0.1972
hmn 0.0007 0.0007 0.0006
hrv 0.2181 0.2081 0.1918
hun 0.4917 0.4644 0.4713
hye 0.0326 0.0353 0.0296
ibo 0.0008 0.0008 0.0007
iku 0.0002 0.0002 0.0002
ile 0.0004 0.0003 0.0003
ina 0.0005 0.0005 0.0005
ind 0.9690 0.9505 1.0124
ipk 0.0001 0.0001 0.0001
isl 0.0412 0.0415 0.0385
ita 2.1221 2.0407 2.0349
jav 0.0025 0.0024 0.0017
jpn 5.1095 5.2018 5.2703
kal 0.0008 0.0009 0.0008
kan 0.0140 0.0142 0.0141
kas 0.0000 0.0000 0.0000
kat 0.0401 0.0433 0.0386
kaz 0.0349 0.0352 0.0332
kha 0.0003 0.0003 0.0003
khm 0.0099 0.0105 0.0107
kin 0.0020 0.0021 0.0021
kir 0.0106 0.0103 0.0095
kor 0.7565 0.7754 0.7145
kur 0.0057 0.0058 0.0060
lao 0.0037 0.0041 0.0038
lat 0.0937 0.0983 0.0998
lav 0.0885 0.0866 0.0859
lif 0.0000 0.0000 0.0000
lin 0.0003 0.0003 0.0002
lit 0.1676 0.1601 0.1626
ltz 0.0034 0.0040 0.0028
lug 0.0001 0.0001 0.0001
mal 0.0226 0.0227 0.0219
mar 0.0249 0.0265 0.0267
mfe 0.0002 0.0002 0.0002
mkd 0.0373 0.0375 0.0337
mlg 0.0042 0.0031 0.0037
mlt 0.0030 0.0036 0.0032
mon 0.0142 0.0148 0.0143
mri 0.0012 0.0014 0.0013
msa 0.0748 0.0675 0.0715
mya 0.0136 0.0136 0.0136
nau 0.0000 0.0000 0.0000
nep 0.0457 0.0454 0.0494
nld 1.8132 1.8755 1.7675
nno 0.0137 0.0140 0.0134
nor 0.3132 0.3213 0.2986
nso 0.0001 0.0001 0.0001
nya 0.0006 0.0006 0.0006
oci 0.0031 0.0033 0.0020
ori 0.0057 0.0052 0.0056
orm 0.0006 0.0005 0.0006
pan 0.0071 0.0074 0.0064
pol 1.7159 1.6602 1.8221
por 2.1796 2.0696 2.2523
pus 0.0038 0.0035 0.0038
que 0.0006 0.0006 0.0006
roh 0.0011 0.0011 0.0010
ron 0.5323 0.4819 0.5160
run 0.0001 0.0001 0.0001
rus 6.1113 6.1083 5.8427
sag 0.0003 0.0004 0.0003
san 0.0030 0.0024 0.0025
sco 0.0006 0.0007 0.0013
sin 0.0086 0.0082 0.0083
slk 0.3928 0.3853 0.3796
slv 0.1343 0.1264 0.1147
smo 0.0007 0.0007 0.0006
sna 0.0007 0.0007 0.0008
snd 0.0017 0.0016 0.0014
som 0.0044 0.0044 0.0046
sot 0.0006 0.0006 0.0006
spa 4.3575 4.3672 4.4619
sqi 0.0462 0.0474 0.0447
srp 0.2141 0.2053 0.2092
ssw 0.0000 0.0000 0.0000
sun 0.0014 0.0015 0.0012
sux 0.0000 0.0000 0.0000
swa 0.0105 0.0104 0.0097
swe 0.6267 0.5874 0.6579
syr 0.0002 0.0003 0.0003
tam 0.0426 0.0396 0.0431
tat 0.0074 0.0080 0.0064
tel 0.0167 0.0174 0.0185
tgk 0.0066 0.0071 0.0054
tgl 0.0081 0.0088 0.0076
tha 0.3743 0.3842 0.3853
tir 0.0007 0.0009 0.0009
ton 0.0001 0.0001 0.0001
tsn 0.0001 0.0001 0.0001
tso 0.0001 0.0001 0.0001
tuk 0.0011 0.0012 0.0011
tur 1.1029 1.1973 1.1516
uig 0.0012 0.0012 0.0012
ukr 0.6857 0.7083 0.6431
urd 0.0274 0.0283 0.0267
uzb 0.0237 0.0250 0.0207
ven 0.0000 0.0000 0.0000
vie 1.0326 1.0549 1.0278
vol 0.0006 0.0008 0.0007
war 0.0024 0.0024 0.0015
wol 0.0001 0.0001 0.0002
xho 0.0008 0.0008 0.0008
yid 0.0017 0.0019 0.0018
yor 0.0011 0.0011 0.0010
zha 0.0001 0.0001 0.0001
zho 5.2056 5.1598 5.0878
zul 0.0014 0.0013 0.0013