Statistics of Common Crawl Monthly Archives

Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives
Latest crawl: CC-MAIN-2024-10

View the Project on GitHub

Distribution of Languages

The language of a document is identified by Compact Language Detector 2 (CLD2). It is able to identify 160 different languages and up to 3 languages per document. The table lists the percentage covered by the primary language of a document (returned first by CLD2). So far, only HTML pages are passed to the language detector. The underlying data including page counts is provided in languages.csv.

crawl CC-MAIN-2023-40 CC-MAIN-2023-50 CC-MAIN-2024-10
language
<unknown> 2.7090 3.2338 2.7619
aar 0.0001 0.0001 0.0001
abk 0.0005 0.0005 0.0004
afr 0.0055 0.0091 0.0081
aka 0.0001 0.0001 0.0001
amh 0.0038 0.0033 0.0031
ara 0.6030 0.5867 0.5875
asm 0.0024 0.0020 0.0020
aym 0.0001 0.0000 0.0001
aze 0.0506 0.0528 0.0603
bak 0.0026 0.0028 0.0025
bel 0.0153 0.0149 0.0158
ben 0.1038 0.0981 0.1025
bih 0.0007 0.0007 0.0006
bis 0.0001 0.0001 0.0001
bod 0.0015 0.0015 0.0018
bos 0.0514 0.0475 0.0480
bre 0.0019 0.0026 0.0017
bul 0.3287 0.2966 0.2877
cat 0.2372 0.2096 0.2053
ceb 0.0015 0.0015 0.0017
ces 1.0611 1.0206 0.9899
chr 0.0000 0.0000 0.0000
cos 0.0021 0.0023 0.0019
crs 0.0001 0.0001 0.0001
cym 0.0097 0.0100 0.0103
dan 0.4475 0.4306 0.4094
deu 5.8355 5.4499 5.3948
div 0.0025 0.0025 0.0026
dzo 0.0001 0.0001 0.0001
ell 0.5861 0.5806 0.5399
eng 46.4328 44.4285 46.4536
epo 0.0082 0.0090 0.0080
est 0.1523 0.1339 0.1290
eus 0.0334 0.0328 0.0329
fao 0.0051 0.0043 0.0041
fas 0.6823 0.6795 0.6666
fij 0.0001 0.0001 0.0001
fin 0.4251 0.3680 0.3577
fra 4.6351 4.3933 4.3263
fry 0.0024 0.0031 0.0024
gla 0.0016 0.0013 0.0013
gle 0.0067 0.0059 0.0061
glg 0.0402 0.0351 0.0335
glv 0.0003 0.0003 0.0003
got 0.0000 0.0000 0.0000
grn 0.0006 0.0006 0.0006
guj 0.0119 0.0114 0.0113
hat 0.0020 0.0017 0.0016
hau 0.0032 0.0021 0.0021
haw 0.0008 0.0006 0.0005
heb 0.1224 0.2091 0.2205
hin 0.1755 0.1749 0.1784
hmn 0.0008 0.0006 0.0005
hrv 0.2249 0.2014 0.1961
hun 0.6247 0.6695 0.5646
hye 0.0321 0.0300 0.0298
ibo 0.0007 0.0005 0.0006
iku 0.0002 0.0002 0.0002
ile 0.0003 0.0003 0.0003
ina 0.0006 0.0005 0.0005
ind 0.8743 0.8596 0.8645
ipk 0.0001 0.0001 0.0000
isl 0.0461 0.0392 0.0399
ita 2.7243 2.5834 2.5737
jav 0.0021 0.0023 0.0020
jpn 4.7475 5.1508 5.0934
kal 0.0008 0.0008 0.0009
kan 0.0139 0.0123 0.0128
kas 0.0000 0.0000 0.0000
kat 0.0409 0.0360 0.0362
kaz 0.0224 0.0221 0.0220
kha 0.0003 0.0003 0.0003
khm 0.0119 0.0097 0.0101
kin 0.0020 0.0019 0.0020
kir 0.0052 0.0056 0.0077
kor 0.6506 0.7288 0.6504
kur 0.0062 0.0056 0.0063
lao 0.0038 0.0036 0.0034
lat 0.1171 0.1169 0.1127
lav 0.1017 0.0849 0.0836
lif 0.0000 0.0000 0.0000
lin 0.0002 0.0002 0.0002
lit 0.2029 0.1553 0.1531
ltz 0.0038 0.0039 0.0034
lug 0.0002 0.0002 0.0002
mal 0.0221 0.0218 0.0215
mar 0.0233 0.0212 0.0234
mfe 0.0001 0.0002 0.0002
mkd 0.0360 0.0334 0.0357
mlg 0.0032 0.0032 0.0028
mlt 0.0048 0.0042 0.0039
mon 0.0172 0.0142 0.0141
mri 0.0014 0.0010 0.0009
msa 0.0819 0.0742 0.0749
mya 0.0113 0.0118 0.0113
nau 0.0000 0.0000 0.0000
nep 0.0398 0.0395 0.0428
nld 2.1566 1.9746 1.9608
nno 0.0154 0.0137 0.0138
nor 0.3735 0.3191 0.3197
nso 0.0001 0.0001 0.0001
nya 0.0007 0.0005 0.0005
oci 0.0027 0.0031 0.0027
ori 0.0049 0.0044 0.0048
orm 0.0005 0.0006 0.0005
pan 0.0061 0.0055 0.0056
pol 1.7413 1.7490 1.7245
por 1.1169 1.7631 1.9278
pus 0.0032 0.0028 0.0029
que 0.0007 0.0006 0.0006
roh 0.0014 0.0013 0.0013
ron 0.5744 0.5690 0.5278
run 0.0001 0.0001 0.0001
rus 5.5709 6.0303 5.8095
sag 0.0001 0.0004 0.0001
san 0.0033 0.0031 0.0030
sco 0.0006 0.0007 0.0006
sin 0.0081 0.0077 0.0080
slk 0.4567 0.3590 0.3608
slv 0.1934 0.1631 0.1510
smo 0.0007 0.0005 0.0005
sna 0.0006 0.0005 0.0005
snd 0.0017 0.0015 0.0016
som 0.0043 0.0041 0.0043
sot 0.0006 0.0005 0.0005
spa 4.6199 4.5391 4.5462
sqi 0.0466 0.0439 0.0450
srp 0.2526 0.2206 0.2150
ssw 0.0000 0.0000 0.0000
sun 0.0022 0.0016 0.0011
sux 0.0000 0.0000 0.0000
swa 0.0084 0.0073 0.0075
swe 0.7013 0.6637 0.6494
syr 0.0003 0.0003 0.0003
tam 0.0432 0.0423 0.0425
tat 0.0058 0.0062 0.0064
tel 0.0182 0.0161 0.0169
tgk 0.0053 0.0052 0.0051
tgl 0.0076 0.0093 0.0084
tha 0.4510 0.4251 0.4101
tir 0.0008 0.0008 0.0008
ton 0.0001 0.0001 0.0001
tsn 0.0001 0.0001 0.0001
tso 0.0000 0.0000 0.0000
tuk 0.0023 0.0025 0.0018
tur 0.8322 0.9804 0.9864
uig 0.0015 0.0013 0.0012
ukr 0.3433 0.3845 0.4295
urd 0.0274 0.0265 0.0284
uzb 0.0169 0.0198 0.0220
ven 0.0000 0.0000 0.0000
vie 1.0570 1.0291 0.9873
vol 0.0005 0.0005 0.0004
war 0.0016 0.0015 0.0019
wol 0.0001 0.0002 0.0001
xho 0.0017 0.0017 0.0017
yid 0.0020 0.0016 0.0015
yor 0.0011 0.0008 0.0008
zha 0.0001 0.0000 0.0000
zho 3.7971 5.0798 4.1666
zul 0.0019 0.0018 0.0018