Statistics of Common Crawl Monthly Archives

Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives

View the Project on GitHub

Distribution of Languages

The language of a document is identified by Compact Language Detector 2 (CLD2). It is able to identify 160 different languages and up to 3 languages per document. The table lists the percentage covered by the primary language of a document (returned first by CLD2). So far, only HTML pages are passed to the language detector. The underlying data including page counts is provided in languages.csv.

crawl CC-MAIN-2019-51 CC-MAIN-2020-05 CC-MAIN-2020-10
language
<other> 0.0001 0.0001 0.0000
<unknown> 2.0022 1.7944 2.3005
aar 0.0002 0.0002 0.0001
abk 0.0004 0.0004 0.0003
afr 0.0121 0.0119 0.0105
aka 0.0001 0.0001 0.0001
amh 0.0022 0.0023 0.0019
ara 0.5835 0.5307 0.5718
asm 0.0010 0.0009 0.0008
aym 0.0001 0.0001 0.0001
aze 0.0541 0.0457 0.0481
bak 0.0018 0.0021 0.0012
bel 0.0217 0.0208 0.0183
ben 0.0707 0.0648 0.0662
bih 0.0006 0.0007 0.0004
bis 0.0001 0.0001 0.0002
blu 0.0005 0.0004 0.0003
bod 0.0020 0.0020 0.0015
bos 0.0440 0.0408 0.0389
bre 0.0032 0.0037 0.0024
bul 0.2937 0.2787 0.2866
cat 0.2820 0.2743 0.2599
ceb 0.0044 0.0030 0.0026
ces 1.0076 1.0588 1.0127
chr 0.0001 0.0001 0.0000
cos 0.0024 0.0024 0.0020
crs 0.0001 0.0001 0.0001
cym 0.0112 0.0115 0.0103
dan 0.4345 0.4565 0.4395
deu 5.4750 5.6726 5.5671
div 0.0015 0.0015 0.0016
dzo 0.0001 0.0001 0.0001
ell 0.5517 0.5432 0.5338
eng 43.5783 44.1754 42.5145
epo 0.0125 0.0098 0.0090
est 0.1704 0.1623 0.1590
eus 0.0380 0.0358 0.0344
fao 0.0045 0.0043 0.0040
fas 0.5490 0.5087 0.5287
fij 0.0001 0.0001 0.0001
fin 0.4079 0.4217 0.3886
fra 4.4824 4.6358 4.5052
fry 0.0028 0.0029 0.0027
gla 0.0012 0.0014 0.0011
gle 0.0058 0.0059 0.0058
glg 0.0421 0.0396 0.0391
glv 0.0003 0.0004 0.0003
grn 0.0008 0.0007 0.0007
guj 0.0087 0.0081 0.0082
hat 0.0015 0.0016 0.0012
hau 0.0015 0.0014 0.0014
haw 0.0005 0.0004 0.0004
heb 0.2862 0.2875 0.2781
hin 0.1083 0.0989 0.1051
hrv 0.2218 0.2079 0.2050
hun 0.5129 0.5145 0.5078
hye 0.0409 0.0371 0.0381
ibo 0.0005 0.0003 0.0004
iku 0.0002 0.0002 0.0002
ile 0.0003 0.0003 0.0003
ina 0.0008 0.0008 0.0006
ind 0.7249 0.7350 0.7053
ipk 0.0001 0.0001 0.0001
isl 0.0448 0.0444 0.0393
ita 2.3067 2.3638 2.2941
jav 0.0024 0.0023 0.0017
jpn 4.4527 4.6593 4.8638
kal 0.0010 0.0011 0.0012
kan 0.0093 0.0082 0.0091
kat 0.0526 0.0469 0.0470
kaz 0.0269 0.0244 0.0239
kha 0.0002 0.0002 0.0002
khm 0.0085 0.0078 0.0080
kin 0.0018 0.0017 0.0015
kir 0.0067 0.0062 0.0059
kor 0.7971 0.7660 0.8203
kur 0.0072 0.0069 0.0068
lao 0.0034 0.0031 0.0025
lat 0.0586 0.0741 0.0853
lav 0.1233 0.1146 0.1148
lin 0.0002 0.0002 0.0002
lit 0.2245 0.2027 0.1941
ltz 0.0048 0.0043 0.0038
lug 0.0002 0.0002 0.0002
mal 0.0185 0.0171 0.0178
mar 0.0149 0.0147 0.0128
mfe 0.0001 0.0001 0.0001
mkd 0.0245 0.0246 0.0244
mlg 0.0040 0.0037 0.0033
mlt 0.0033 0.0032 0.0029
mon 0.0200 0.0179 0.0178
mri 0.0012 0.0013 0.0010
msa 0.1509 0.1538 0.1112
mya 0.0151 0.0125 0.0106
nau 0.0000 0.0000 0.0000
nep 0.0161 0.0163 0.0153
nld 1.7218 1.8009 1.7341
nno 0.0229 0.0214 0.0178
nor 0.3390 0.3456 0.3225
nso 0.0002 0.0002 0.0001
nya 0.0002 0.0002 0.0002
oci 0.0044 0.0043 0.0031
ori 0.0022 0.0020 0.0019
orm 0.0004 0.0003 0.0002
pan 0.0059 0.0055 0.0053
pol 1.6601 1.6774 1.6895
por 2.1156 2.1660 2.1373
pus 0.0030 0.0028 0.0025
que 0.0009 0.0011 0.0005
roh 0.0013 0.0013 0.0022
ron 0.5463 0.5472 0.5590
run 0.0001 0.0001 0.0001
rus 6.5412 6.6710 6.5662
san 0.0030 0.0029 0.0029
sco 0.0005 0.0005 0.0006
sin 0.0068 0.0064 0.0062
slk 0.3078 0.3367 0.3526
slv 0.1676 0.1594 0.1616
smo 0.0003 0.0003 0.0002
sna 0.0002 0.0002 0.0002
snd 0.0007 0.0009 0.0007
som 0.0035 0.0035 0.0034
sot 0.0006 0.0003 0.0003
spa 4.2281 4.2994 4.1250
sqi 0.0434 0.0381 0.0388
srp 0.2449 0.2294 0.2241
ssw 0.0000 0.0000 0.0000
sun 0.0013 0.0013 0.0010
swa 0.0076 0.0073 0.0061
swe 0.7688 0.7824 0.7517
syr 0.0005 0.0004 0.0005
tam 0.0470 0.0459 0.0420
tat 0.0062 0.0067 0.0054
tel 0.0172 0.0171 0.0157
tgk 0.0059 0.0053 0.0044
tgl 0.0070 0.0065 0.0054
tha 0.3363 0.3272 0.3272
tir 0.0009 0.0007 0.0006
ton 0.0001 0.0001 0.0001
tsn 0.0002 0.0003 0.0003
tso 0.0002 0.0001 0.0000
tuk 0.0016 0.0014 0.0013
tur 0.8206 0.8137 0.8415
uig 0.0014 0.0014 0.0016
ukr 0.3530 0.3472 0.3623
urd 0.0257 0.0241 0.0208
uzb 0.0203 0.0188 0.0168
ven 0.0000 0.0000 0.0000
vie 0.7528 0.7435 0.7128
vol 0.0014 0.0011 0.0015
war 0.0043 0.0043 0.0028
wol 0.0002 0.0002 0.0002
xho 0.0003 0.0003 0.0004
yid 0.0027 0.0027 0.0015
yor 0.0010 0.0010 0.0007
zha 0.0001 0.0001 0.0000
zho 8.3698 7.1851 8.9495
zul 0.0011 0.0006 0.0006