Statistics of Common Crawl Monthly Archives

Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives
Latest crawl: CC-MAIN-2025-13

View the Project on GitHub

Distribution of Languages

The language of a document is identified by Compact Language Detector 2 (CLD2). It is able to identify 160 different languages and up to 3 languages per document. The table lists the percentage covered by the primary language of a document (returned first by CLD2). So far, only HTML pages are passed to the language detector. The underlying data including page counts is provided in languages.csv.

crawl CC-MAIN-2025-05 CC-MAIN-2025-08 CC-MAIN-2025-13
language
<unknown> 2.6949 3.0224 3.0027
aar 0.0001 0.0001 0.0001
abk 0.0005 0.0006 0.0005
afr 0.0087 0.0099 0.0095
aka 0.0001 0.0002 0.0001
amh 0.0028 0.0031 0.0033
ara 0.6640 0.6856 0.6837
asm 0.0025 0.0025 0.0023
aym 0.0001 0.0000 0.0001
aze 0.0572 0.0630 0.0612
bak 0.0026 0.0029 0.0032
bel 0.0143 0.0149 0.0165
ben 0.1089 0.1147 0.1107
bih 0.0005 0.0005 0.0005
bis 0.0001 0.0001 0.0001
bod 0.0014 0.0015 0.0015
bos 0.0520 0.0571 0.0538
bre 0.0018 0.0029 0.0023
bul 0.2985 0.3103 0.3027
cat 0.1876 0.1849 0.1767
ceb 0.0015 0.0017 0.0018
ces 1.0225 1.0083 1.0380
chr 0.0000 0.0000 0.0000
cos 0.0019 0.0018 0.0021
crs 0.0001 0.0001 0.0001
cym 0.0109 0.0107 0.0120
dan 0.4850 0.4671 0.4630
deu 5.5857 5.4039 5.1154
div 0.0022 0.0025 0.0024
dzo 0.0002 0.0002 0.0002
ell 0.5740 0.5833 0.5902
eng 43.3694 43.3794 43.9338
epo 0.0074 0.0074 0.0085
est 0.1345 0.1306 0.1308
eus 0.0347 0.0346 0.0328
fao 0.0045 0.0043 0.0044
fas 0.7250 0.7274 0.7350
fij 0.0001 0.0001 0.0001
fin 0.3678 0.3563 0.3616
fra 4.5221 4.2950 4.2269
fry 0.0020 0.0027 0.0022
gla 0.0013 0.0013 0.0013
gle 0.0066 0.0074 0.0074
glg 0.0320 0.0297 0.0305
glv 0.0005 0.0004 0.0009
got 0.0000 0.0000 0.0000
grn 0.0007 0.0007 0.0007
guj 0.0120 0.0124 0.0125
hat 0.0018 0.0020 0.0019
hau 0.0026 0.0027 0.0032
haw 0.0006 0.0006 0.0006
heb 0.2701 0.2563 0.2571
hin 0.1951 0.2040 0.2027
hmn 0.0005 0.0006 0.0006
hrv 0.2238 0.2258 0.2295
hun 0.5023 0.5021 0.4990
hye 0.0318 0.0327 0.0328
ibo 0.0006 0.0006 0.0006
iku 0.0002 0.0002 0.0002
ile 0.0004 0.0003 0.0003
ina 0.0006 0.0005 0.0005
ind 1.1676 1.1845 1.2009
ipk 0.0001 0.0001 0.0001
isl 0.0423 0.0389 0.0388
ita 2.4152 2.3997 2.4181
jav 0.0022 0.0023 0.0023
jpn 5.4046 5.1641 4.8624
kal 0.0008 0.0008 0.0008
kan 0.0125 0.0135 0.0138
kas 0.0000 0.0000 0.0000
kat 0.0396 0.0415 0.0444
kaz 0.0272 0.0298 0.0318
kha 0.0004 0.0004 0.0003
khm 0.0104 0.0109 0.0106
kin 0.0019 0.0021 0.0023
kir 0.0097 0.0091 0.0111
kor 0.7183 0.8078 0.8028
kur 0.0063 0.0068 0.0069
lao 0.0033 0.0040 0.0039
lat 0.1195 0.1060 0.1061
lav 0.0888 0.0878 0.0854
lif 0.0000 0.0000 0.0000
lin 0.0003 0.0003 0.0003
lit 0.1659 0.1627 0.1659
ltz 0.0038 0.0038 0.0041
lug 0.0002 0.0002 0.0001
mal 0.0228 0.0251 0.0253
mar 0.0258 0.0278 0.0290
mfe 0.0001 0.0001 0.0001
mkd 0.0361 0.0380 0.0387
mlg 0.0028 0.0035 0.0041
mlt 0.0030 0.0032 0.0032
mon 0.0146 0.0162 0.0153
mri 0.0013 0.0013 0.0013
msa 0.0781 0.0785 0.0774
mya 0.0129 0.0137 0.0148
nau 0.0000 0.0000 0.0000
nep 0.0519 0.0552 0.0553
nld 1.9167 1.7941 1.7982
nno 0.0162 0.0167 0.0135
nor 0.3142 0.3042 0.3119
nso 0.0001 0.0001 0.0001
nya 0.0005 0.0005 0.0005
oci 0.0034 0.0033 0.0037
ori 0.0056 0.0060 0.0061
orm 0.0006 0.0005 0.0007
pan 0.0069 0.0070 0.0068
pol 1.8248 1.7470 1.6923
por 2.3259 2.3274 2.3679
pus 0.0030 0.0031 0.0041
que 0.0006 0.0005 0.0005
roh 0.0010 0.0010 0.0010
ron 0.6189 0.6304 0.6522
run 0.0000 0.0001 0.0001
rus 6.0460 6.0982 5.9294
sag 0.0006 0.0001 0.0007
san 0.0030 0.0032 0.0032
sco 0.0005 0.0006 0.0006
sin 0.0098 0.0094 0.0092
slk 0.3617 0.3568 0.3684
slv 0.1504 0.1509 0.1552
smo 0.0006 0.0006 0.0006
sna 0.0007 0.0006 0.0007
snd 0.0014 0.0016 0.0016
som 0.0046 0.0052 0.0048
sot 0.0006 0.0006 0.0005
spa 4.6417 4.5461 4.4426
sqi 0.0452 0.0487 0.0492
srp 0.2174 0.2152 0.2157
ssw 0.0000 0.0000 0.0000
sun 0.0010 0.0010 0.0012
sux 0.0000 0.0000 0.0000
swa 0.0085 0.0083 0.0095
swe 0.6547 0.6415 0.6621
syr 0.0003 0.0004 0.0004
tam 0.0422 0.0457 0.0437
tat 0.0069 0.0081 0.0082
tel 0.0180 0.0198 0.0195
tgk 0.0063 0.0065 0.0069
tgl 0.0078 0.0083 0.0087
tha 0.4554 0.5037 0.4533
tir 0.0008 0.0007 0.0007
ton 0.0001 0.0001 0.0001
tsn 0.0001 0.0001 0.0001
tso 0.0001 0.0001 0.0001
tuk 0.0017 0.0020 0.0022
tur 1.1750 1.1411 1.1405
uig 0.0010 0.0012 0.0012
ukr 0.6046 0.6075 0.6240
urd 0.0276 0.0303 0.0300
uzb 0.0199 0.0226 0.0228
ven 0.0001 0.0000 0.0000
vie 1.0427 1.0551 1.0498
vol 0.0005 0.0005 0.0005
war 0.0019 0.0023 0.0021
wol 0.0001 0.0001 0.0001
xho 0.0008 0.0008 0.0007
yid 0.0014 0.0017 0.0015
yor 0.0009 0.0008 0.0008
zha 0.0001 0.0002 0.0001
zho 4.6719 5.0942 5.4135
zul 0.0012 0.0013 0.0011