Statistics of Common Crawl Monthly Archives

Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives

View the Project on GitHub

Distribution of Languages

The language of a document is identified by Compact Language Detector 2 (CLD2). It is able to identify 160 different languages and up to 3 languages per document. The table lists the percentage covered by the primary language of a document (returned first by CLD2). So far, only HTML pages are passed to the language detector.

crawl CC-MAIN-2018-34
language
<other> 0.0000
aar 0.0002
abk 0.0002
afr 0.0077
aka 0.0001
amh 0.0022
ara 0.7106
asm 0.0005
aym 0.0001
aze 0.0361
bak 0.0012
bel 0.0126
ben 0.0482
bih 0.0006
bis 0.0001
blu 0.0007
bod 0.0011
bos 0.0298
bre 0.0014
bul 0.2770
cat 0.1811
ceb 0.0012
ces 1.0815
chr 0.0000
cos 0.0014
crs 0.0001
cym 0.0065
dan 0.3870
deu 5.2847
div 0.0007
dzo 0.0001
ell 0.5089
eng 46.6226
epo 0.0043
est 0.1040
eus 0.0192
fao 0.0028
fas 0.6730
fij 0.0001
fin 0.3755
fra 4.5931
fry 0.0022
gla 0.0015
gle 0.0039
glg 0.0233
glv 0.0003
got 0.0000
grn 0.0009
guj 0.0052
hat 0.0013
hau 0.0016
haw 0.0003
heb 0.1706
hin 0.0729
hrv 0.1421
hun 0.5155
hye 0.0239
ibo 0.0002
iku 0.0001
ile 0.0003
ina 0.0008
ind 0.7861
ipk 0.0001
isl 0.0324
ita 2.0861
jav 0.0012
jpn 5.6766
kal 0.0006
kan 0.0063
kat 0.0267
kaz 0.0143
kha 0.0002
khm 0.0052
kin 0.0018
kir 0.0034
kor 0.5940
kur 0.0043
lao 0.0020
lat 0.0352
lav 0.0769
lin 0.0002
lit 0.1366
ltz 0.0021
lug 0.0001
mal 0.0136
mar 0.0075
mfe 0.0001
mkd 0.0175
mlg 0.0021
mlt 0.0020
mon 0.0128
mri 0.0006
msa 0.1048
mya 0.0084
nau 0.0000
nep 0.0115
nld 1.4527
nno 0.0131
nor 0.3097
nso 0.0001
nya 0.0005
oci 0.0026
ori 0.0018
orm 0.0003
pan 0.0031
pol 1.7673
por 2.1048
pus 0.0023
que 0.0003
roh 0.0013
ron 0.5437
run 0.0001
rus 9.5047
sag 0.0000
san 0.0018
sco 0.0008
sin 0.0056
slk 0.2407
slv 0.1018
smo 0.0003
sna 0.0003
snd 0.0005
som 0.0045
sot 0.0002
spa 4.3270
sqi 0.0301
srp 0.1546
ssw 0.0001
sun 0.0006
swa 0.0067
swe 0.7680
syr 0.0003
tam 0.0317
tat 0.0051
tel 0.0088
tgk 0.0025
tgl 0.0083
tha 0.3376
tir 0.0007
ton 0.0001
tsn 0.0001
tso 0.0001
tuk 0.0009
tur 0.9263
uig 0.0009
ukr 0.3868
urd 0.0211
uzb 0.0099
ven 0.0000
vie 0.7932
vol 0.0006
war 0.0022
wol 0.0001
xho 0.0004
yid 0.0013
yor 0.0003
zha 0.0000
zho 4.4900
zul 0.0004