Statistics of Common Crawl Monthly Archives

Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives
Latest crawl: CC-MAIN-2025-38

View the Project on GitHub

Distribution of Languages

The language of a document is identified by Compact Language Detector 2 (CLD2). It is able to identify 160 different languages and up to 3 languages per document. The table lists the percentage covered by the primary language of a document (returned first by CLD2). So far, only HTML pages are passed to the language detector. The underlying data including page counts is provided in languages.csv.

crawl CC-MAIN-2025-30 CC-MAIN-2025-33 CC-MAIN-2025-38
language
<unknown> 2.9989 2.7925 2.7649
aar 0.0001 0.0001 0.0001
abk 0.0005 0.0006 0.0006
afr 0.0083 0.0086 0.0082
aka 0.0002 0.0002 0.0001
amh 0.0033 0.0032 0.0033
ara 0.6825 0.6383 0.6549
asm 0.0024 0.0025 0.0025
aym 0.0001 0.0001 0.0001
aze 0.0593 0.0576 0.0624
bak 0.0014 0.0015 0.0014
bel 0.0156 0.0150 0.0153
ben 0.1041 0.1040 0.1093
bih 0.0005 0.0005 0.0005
bis 0.0001 0.0001 0.0001
bod 0.0013 0.0013 0.0009
bos 0.0521 0.0530 0.0557
bre 0.0015 0.0033 0.0016
bul 0.2590 0.2584 0.2840
cat 0.1775 0.1732 0.1741
ceb 0.0017 0.0019 0.0021
ces 1.0122 1.0202 0.9881
chr 0.0000 0.0000 0.0000
cos 0.0020 0.0022 0.0022
crs 0.0001 0.0001 0.0001
cym 0.0106 0.0100 0.0094
dan 0.4678 0.4822 0.4432
deu 5.5837 5.8191 6.0060
div 0.0023 0.0024 0.0025
dzo 0.0001 0.0001 0.0001
ell 0.5014 0.5255 0.4898
eng 44.6146 44.2668 44.3285
epo 0.0079 0.0078 0.0073
est 0.1191 0.1182 0.1170
eus 0.0344 0.0308 0.0306
fao 0.0047 0.0047 0.0045
fas 0.7373 0.7886 0.7814
fij 0.0001 0.0001 0.0001
fin 0.3354 0.3422 0.3184
fra 4.0933 4.4559 4.4605
fry 0.0020 0.0017 0.0032
gla 0.0016 0.0015 0.0014
gle 0.0068 0.0070 0.0069
glg 0.0290 0.0272 0.0279
glv 0.0004 0.0004 0.0004
got 0.0000 0.0000 0.0000
grn 0.0007 0.0008 0.0008
guj 0.0111 0.0106 0.0110
hat 0.0019 0.0025 0.0017
hau 0.0027 0.0024 0.0027
haw 0.0007 0.0007 0.0008
heb 0.2385 0.2450 0.2285
hin 0.1996 0.1927 0.1930
hmn 0.0006 0.0007 0.0007
hrv 0.2103 0.2181 0.2081
hun 0.4779 0.4917 0.4644
hye 0.0326 0.0326 0.0353
ibo 0.0008 0.0008 0.0008
iku 0.0002 0.0002 0.0002
ile 0.0003 0.0004 0.0003
ina 0.0004 0.0005 0.0005
ind 0.9542 0.9690 0.9505
ipk 0.0001 0.0001 0.0001
isl 0.0389 0.0412 0.0415
ita 2.4126 2.1221 2.0407
jav 0.0019 0.0025 0.0024
jpn 5.2783 5.1095 5.2018
kal 0.0008 0.0008 0.0009
kan 0.0147 0.0140 0.0142
kas 0.0000 0.0000 0.0000
kat 0.0393 0.0401 0.0433
kaz 0.0331 0.0349 0.0352
kha 0.0003 0.0003 0.0003
khm 0.0109 0.0099 0.0105
kin 0.0020 0.0020 0.0021
kir 0.0101 0.0106 0.0103
kor 0.7753 0.7565 0.7754
kur 0.0060 0.0057 0.0058
lao 0.0036 0.0037 0.0041
lat 0.0868 0.0937 0.0983
lav 0.0895 0.0885 0.0866
lif 0.0000 0.0000 0.0000
lin 0.0003 0.0003 0.0003
lit 0.1624 0.1676 0.1601
ltz 0.0035 0.0034 0.0040
lug 0.0002 0.0001 0.0001
mal 0.0237 0.0226 0.0227
mar 0.0267 0.0249 0.0265
mfe 0.0001 0.0002 0.0002
mkd 0.0359 0.0373 0.0375
mlg 0.0037 0.0042 0.0031
mlt 0.0032 0.0030 0.0036
mon 0.0149 0.0142 0.0148
mri 0.0013 0.0012 0.0014
msa 0.0727 0.0748 0.0675
mya 0.0128 0.0136 0.0136
nau 0.0000 0.0000 0.0000
nep 0.0480 0.0457 0.0454
nld 1.7131 1.8132 1.8755
nno 0.0134 0.0137 0.0140
nor 0.3075 0.3132 0.3213
nso 0.0001 0.0001 0.0001
nya 0.0006 0.0006 0.0006
oci 0.0030 0.0031 0.0033
ori 0.0057 0.0057 0.0052
orm 0.0005 0.0006 0.0005
pan 0.0067 0.0071 0.0074
pol 1.6697 1.7159 1.6602
por 2.1612 2.1796 2.0696
pus 0.0033 0.0038 0.0035
que 0.0006 0.0006 0.0006
roh 0.0009 0.0011 0.0011
ron 0.4942 0.5323 0.4819
run 0.0001 0.0001 0.0001
rus 5.8351 6.1113 6.1083
sag 0.0003 0.0003 0.0004
san 0.0029 0.0030 0.0024
sco 0.0007 0.0006 0.0007
sin 0.0083 0.0086 0.0082
slk 0.3697 0.3928 0.3853
slv 0.1274 0.1343 0.1264
smo 0.0007 0.0007 0.0007
sna 0.0007 0.0007 0.0007
snd 0.0018 0.0017 0.0016
som 0.0045 0.0044 0.0044
sot 0.0006 0.0006 0.0006
spa 4.2873 4.3575 4.3672
sqi 0.0455 0.0462 0.0474
srp 0.2116 0.2141 0.2053
ssw 0.0000 0.0000 0.0000
sun 0.0013 0.0014 0.0015
sux 0.0000 0.0000 0.0000
swa 0.0099 0.0105 0.0104
swe 0.6121 0.6267 0.5874
syr 0.0003 0.0002 0.0003
tam 0.0428 0.0426 0.0396
tat 0.0077 0.0074 0.0080
tel 0.0177 0.0167 0.0174
tgk 0.0066 0.0066 0.0071
tgl 0.0081 0.0081 0.0088
tha 0.3682 0.3743 0.3842
tir 0.0007 0.0007 0.0009
ton 0.0001 0.0001 0.0001
tsn 0.0001 0.0001 0.0001
tso 0.0001 0.0001 0.0001
tuk 0.0025 0.0011 0.0012
tur 1.1006 1.1029 1.1973
uig 0.0012 0.0012 0.0012
ukr 0.6469 0.6857 0.7083
urd 0.0290 0.0274 0.0283
uzb 0.0238 0.0237 0.0250
ven 0.0000 0.0000 0.0000
vie 0.9519 1.0326 1.0549
vol 0.0006 0.0006 0.0008
war 0.0025 0.0024 0.0024
wol 0.0001 0.0001 0.0001
xho 0.0009 0.0008 0.0008
yid 0.0017 0.0017 0.0019
yor 0.0011 0.0011 0.0011
zha 0.0001 0.0001 0.0001
zho 5.5983 5.2056 5.1598
zul 0.0014 0.0014 0.0013