Statistics of Common Crawl Monthly Archives

Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives

View the Project on GitHub

Distribution of Languages

The language of a document is identified by Compact Language Detector 2 (CLD2). It is able to identify 160 different languages and up to 3 languages per document. The table lists the percentage covered by the primary language of a document (returned first by CLD2). So far, only HTML pages are passed to the language detector. The underlying data including page counts is provided in languages.csv.

crawl CC-MAIN-2020-10 CC-MAIN-2020-16 CC-MAIN-2020-24
language
<other> 0.0000 0.0000 0.0000
<unknown> 2.3005 2.4684 2.1541
aar 0.0001 0.0002 0.0001
abk 0.0003 0.0004 0.0004
afr 0.0105 0.0114 0.0119
aka 0.0001 0.0001 0.0000
amh 0.0019 0.0022 0.0026
ara 0.5718 0.5002 0.6194
asm 0.0008 0.0007 0.0009
aym 0.0001 0.0001 0.0001
aze 0.0481 0.0524 0.0688
bak 0.0012 0.0013 0.0014
bel 0.0183 0.0194 0.0208
ben 0.0662 0.0684 0.0876
bih 0.0004 0.0006 0.0006
bis 0.0002 0.0001 0.0001
blu 0.0003 0.0004 0.0005
bod 0.0015 0.0017 0.0017
bos 0.0389 0.0409 0.0494
bre 0.0024 0.0030 0.0032
bul 0.2866 0.3331 0.3812
cat 0.2599 0.2893 0.3034
ceb 0.0026 0.0015 0.0024
ces 1.0127 1.0672 0.9951
chr 0.0000 0.0001 0.0001
cos 0.0020 0.0024 0.0024
crs 0.0001 0.0001 0.0001
cym 0.0103 0.0122 0.0130
dan 0.4395 0.4761 0.4584
deu 5.5671 5.7597 5.5602
div 0.0016 0.0016 0.0021
dzo 0.0001 0.0001 0.0001
ell 0.5338 0.5131 0.5695
eng 42.5145 42.6305 42.3306
epo 0.0090 0.0080 0.0085
est 0.1590 0.1846 0.2000
eus 0.0344 0.0388 0.0428
fao 0.0040 0.0049 0.0054
fas 0.5287 0.5274 0.5904
fij 0.0001 0.0001 0.0001
fin 0.3886 0.4149 0.4012
fra 4.5052 4.5498 4.4114
fry 0.0027 0.0030 0.0032
gla 0.0011 0.0014 0.0014
gle 0.0058 0.0068 0.0064
glg 0.0391 0.0446 0.0484
glv 0.0003 0.0005 0.0004
got 0.0000 0.0000 0.0000
grn 0.0007 0.0007 0.0007
guj 0.0082 0.0074 0.0087
hat 0.0012 0.0013 0.0017
hau 0.0014 0.0013 0.0016
haw 0.0004 0.0004 0.0004
heb 0.2781 0.3252 0.3333
hin 0.1051 0.1020 0.1165
hrv 0.2050 0.2362 0.2728
hun 0.5078 0.5136 0.5320
hye 0.0381 0.0381 0.0452
ibo 0.0004 0.0004 0.0004
iku 0.0002 0.0002 0.0002
ile 0.0003 0.0003 0.0003
ina 0.0006 0.0008 0.0008
ind 0.7053 0.7239 0.7592
ipk 0.0001 0.0001 0.0001
isl 0.0393 0.0472 0.0485
ita 2.2941 2.3491 2.3351
jav 0.0017 0.0021 0.0023
jpn 4.8638 4.7897 4.7113
kal 0.0012 0.0012 0.0013
kan 0.0091 0.0077 0.0093
kat 0.0470 0.0495 0.0608
kaz 0.0239 0.0254 0.0302
kha 0.0002 0.0002 0.0002
khm 0.0080 0.0074 0.0096
kin 0.0015 0.0014 0.0018
kir 0.0059 0.0062 0.0073
kor 0.8203 0.7323 0.8566
kur 0.0068 0.0068 0.0082
lao 0.0025 0.0026 0.0028
lat 0.0853 0.0981 0.1117
lav 0.1148 0.1373 0.1535
lin 0.0002 0.0003 0.0004
lit 0.1941 0.2318 0.2706
ltz 0.0038 0.0046 0.0045
lug 0.0002 0.0002 0.0003
mal 0.0178 0.0162 0.0205
mar 0.0128 0.0125 0.0138
mfe 0.0001 0.0001 0.0001
mkd 0.0244 0.0235 0.0275
mlg 0.0033 0.0031 0.0036
mlt 0.0029 0.0030 0.0035
mon 0.0178 0.0195 0.0231
mri 0.0010 0.0010 0.0011
msa 0.1112 0.1178 0.1238
mya 0.0106 0.0111 0.0116
nau 0.0000 0.0000 0.0000
nep 0.0153 0.0157 0.0207
nld 1.7341 1.8837 1.7901
nno 0.0178 0.0210 0.0206
nor 0.3225 0.3432 0.3477
nso 0.0001 0.0001 0.0001
nya 0.0002 0.0003 0.0003
oci 0.0031 0.0031 0.0034
ori 0.0019 0.0018 0.0027
orm 0.0002 0.0003 0.0003
pan 0.0053 0.0054 0.0065
pol 1.6895 1.7221 1.7692
por 2.1373 2.1227 2.2330
pus 0.0025 0.0025 0.0032
que 0.0005 0.0008 0.0008
roh 0.0022 0.0020 0.0022
ron 0.5590 0.5633 0.6047
run 0.0001 0.0001 0.0001
rus 6.5662 6.9275 6.7749
san 0.0029 0.0030 0.0033
sco 0.0006 0.0006 0.0006
sin 0.0062 0.0065 0.0079
slk 0.3526 0.3179 0.3068
slv 0.1616 0.1922 0.2189
smo 0.0002 0.0005 0.0005
sna 0.0002 0.0002 0.0004
snd 0.0007 0.0009 0.0012
som 0.0034 0.0029 0.0048
sot 0.0003 0.0004 0.0004
spa 4.1250 4.0854 4.2124
sqi 0.0388 0.0391 0.0509
srp 0.2241 0.2553 0.2902
ssw 0.0000 0.0000 0.0000
sun 0.0010 0.0012 0.0011
swa 0.0061 0.0073 0.0079
swe 0.7517 0.7807 0.7675
syr 0.0005 0.0005 0.0005
tam 0.0420 0.0405 0.0450
tat 0.0054 0.0055 0.0064
tel 0.0157 0.0159 0.0184
tgk 0.0044 0.0052 0.0058
tgl 0.0054 0.0064 0.0066
tha 0.3272 0.3011 0.3506
tir 0.0006 0.0008 0.0006
ton 0.0001 0.0001 0.0001
tsn 0.0003 0.0003 0.0003
tso 0.0000 0.0001 0.0000
tuk 0.0013 0.0015 0.0018
tur 0.8415 0.8200 0.9316
uig 0.0016 0.0017 0.0018
ukr 0.3623 0.3638 0.3858
urd 0.0208 0.0203 0.0246
uzb 0.0168 0.0185 0.0218
ven 0.0000 0.0000 0.0000
vie 0.7128 0.7248 0.7698
vol 0.0015 0.0014 0.0012
war 0.0028 0.0031 0.0031
wol 0.0002 0.0002 0.0001
xho 0.0004 0.0004 0.0005
yid 0.0015 0.0018 0.0020
yor 0.0007 0.0009 0.0009
zha 0.0000 0.0001 0.0001
zho 8.9495 7.6510 7.6740
zul 0.0006 0.0006 0.0008