Statistics of Common Crawl Monthly Archives

Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives
Latest crawl: CC-MAIN-2025-18

View the Project on GitHub

Distribution of Languages

The language of a document is identified by Compact Language Detector 2 (CLD2). It is able to identify 160 different languages and up to 3 languages per document. The table lists the percentage covered by the primary language of a document (returned first by CLD2). So far, only HTML pages are passed to the language detector. The underlying data including page counts is provided in languages.csv.

crawl CC-MAIN-2025-08 CC-MAIN-2025-13 CC-MAIN-2025-18
language
<unknown> 3.0224 3.0027 2.9817
aar 0.0001 0.0001 0.0001
abk 0.0006 0.0005 0.0005
afr 0.0099 0.0095 0.0100
aka 0.0002 0.0001 0.0001
amh 0.0031 0.0033 0.0031
ara 0.6856 0.6837 0.6722
asm 0.0025 0.0023 0.0025
aym 0.0000 0.0001 0.0001
aze 0.0630 0.0612 0.0602
bak 0.0029 0.0032 0.0019
bel 0.0149 0.0165 0.0148
ben 0.1147 0.1107 0.1064
bih 0.0005 0.0005 0.0006
bis 0.0001 0.0001 0.0001
bod 0.0015 0.0015 0.0013
bos 0.0571 0.0538 0.0559
bre 0.0029 0.0023 0.0027
bul 0.3103 0.3027 0.2964
cat 0.1849 0.1767 0.1893
ceb 0.0017 0.0018 0.0019
ces 1.0083 1.0380 1.0479
chr 0.0000 0.0000 0.0000
cos 0.0018 0.0021 0.0019
crs 0.0001 0.0001 0.0001
cym 0.0107 0.0120 0.0114
dan 0.4671 0.4630 0.4792
deu 5.4039 5.1154 5.5691
div 0.0025 0.0024 0.0025
dzo 0.0002 0.0002 0.0002
ell 0.5833 0.5902 0.5651
eng 43.3794 43.9338 43.9499
epo 0.0074 0.0085 0.0073
est 0.1306 0.1308 0.1324
eus 0.0346 0.0328 0.0335
fao 0.0043 0.0044 0.0045
fas 0.7274 0.7350 0.7087
fij 0.0001 0.0001 0.0001
fin 0.3563 0.3616 0.3649
fra 4.2950 4.2269 4.3271
fry 0.0027 0.0022 0.0026
gla 0.0013 0.0013 0.0012
gle 0.0074 0.0074 0.0073
glg 0.0297 0.0305 0.0335
glv 0.0004 0.0009 0.0004
got 0.0000 0.0000 0.0000
grn 0.0007 0.0007 0.0006
guj 0.0124 0.0125 0.0111
hat 0.0020 0.0019 0.0019
hau 0.0027 0.0032 0.0029
haw 0.0006 0.0006 0.0007
heb 0.2563 0.2571 0.2654
hin 0.2040 0.2027 0.2004
hmn 0.0006 0.0006 0.0006
hrv 0.2258 0.2295 0.2339
hun 0.5021 0.4990 0.5082
hye 0.0327 0.0328 0.0328
ibo 0.0006 0.0006 0.0007
iku 0.0002 0.0002 0.0003
ile 0.0003 0.0003 0.0006
ina 0.0005 0.0005 0.0005
ind 1.1845 1.2009 1.1759
ipk 0.0001 0.0001 0.0001
isl 0.0389 0.0388 0.0391
ita 2.3997 2.4181 2.4060
jav 0.0023 0.0023 0.0022
jpn 5.1641 4.8624 4.9152
kal 0.0008 0.0008 0.0008
kan 0.0135 0.0138 0.0130
kas 0.0000 0.0000 0.0000
kat 0.0415 0.0444 0.0435
kaz 0.0298 0.0318 0.0317
kha 0.0004 0.0003 0.0003
khm 0.0109 0.0106 0.0102
kin 0.0021 0.0023 0.0020
kir 0.0091 0.0111 0.0096
kor 0.8078 0.8028 0.7865
kur 0.0068 0.0069 0.0068
lao 0.0040 0.0039 0.0035
lat 0.1060 0.1061 0.1025
lav 0.0878 0.0854 0.0896
lif 0.0000 0.0000 0.0000
lin 0.0003 0.0003 0.0002
lit 0.1627 0.1659 0.1673
ltz 0.0038 0.0041 0.0038
lug 0.0002 0.0001 0.0002
mal 0.0251 0.0253 0.0233
mar 0.0278 0.0290 0.0273
mfe 0.0001 0.0001 0.0001
mkd 0.0380 0.0387 0.0381
mlg 0.0035 0.0041 0.0050
mlt 0.0032 0.0032 0.0031
mon 0.0162 0.0153 0.0154
mri 0.0013 0.0013 0.0014
msa 0.0785 0.0774 0.0767
mya 0.0137 0.0148 0.0132
nau 0.0000 0.0000 0.0000
nep 0.0552 0.0553 0.0522
nld 1.7941 1.7982 1.8083
nno 0.0167 0.0135 0.0143
nor 0.3042 0.3119 0.3135
nso 0.0001 0.0001 0.0001
nya 0.0005 0.0005 0.0004
oci 0.0033 0.0037 0.0033
ori 0.0060 0.0061 0.0058
orm 0.0005 0.0007 0.0004
pan 0.0070 0.0068 0.0068
pol 1.7470 1.6923 1.8744
por 2.3274 2.3679 2.3369
pus 0.0031 0.0041 0.0039
que 0.0005 0.0005 0.0005
roh 0.0010 0.0010 0.0011
ron 0.6304 0.6522 0.6374
run 0.0001 0.0001 0.0001
rus 6.0982 5.9294 5.7614
sag 0.0001 0.0007 0.0001
san 0.0032 0.0032 0.0032
sco 0.0006 0.0006 0.0006
sin 0.0094 0.0092 0.0086
slk 0.3568 0.3684 0.3706
slv 0.1509 0.1552 0.1605
smo 0.0006 0.0006 0.0007
sna 0.0006 0.0007 0.0007
snd 0.0016 0.0016 0.0015
som 0.0052 0.0048 0.0045
sot 0.0006 0.0005 0.0005
spa 4.5461 4.4426 4.5422
sqi 0.0487 0.0492 0.0486
srp 0.2152 0.2157 0.2133
ssw 0.0000 0.0000 0.0000
sun 0.0010 0.0012 0.0011
sux 0.0000 0.0000 0.0000
swa 0.0083 0.0095 0.0102
swe 0.6415 0.6621 0.6736
syr 0.0004 0.0004 0.0003
tam 0.0457 0.0437 0.0447
tat 0.0081 0.0082 0.0075
tel 0.0198 0.0195 0.0213
tgk 0.0065 0.0069 0.0065
tgl 0.0083 0.0087 0.0084
tha 0.5037 0.4533 0.4269
tir 0.0007 0.0007 0.0008
ton 0.0001 0.0001 0.0001
tsn 0.0001 0.0001 0.0002
tso 0.0001 0.0001 0.0000
tuk 0.0020 0.0022 0.0021
tur 1.1411 1.1405 1.1274
uig 0.0012 0.0012 0.0014
ukr 0.6075 0.6240 0.6079
urd 0.0303 0.0300 0.0289
uzb 0.0226 0.0228 0.0217
ven 0.0000 0.0000 0.0000
vie 1.0551 1.0498 1.0213
vol 0.0005 0.0005 0.0005
war 0.0023 0.0021 0.0020
wol 0.0001 0.0001 0.0001
xho 0.0008 0.0007 0.0007
yid 0.0017 0.0015 0.0014
yor 0.0008 0.0008 0.0008
zha 0.0002 0.0001 0.0001
zho 5.0942 5.4135 4.8778
zul 0.0013 0.0011 0.0011