Statistics of Common Crawl Monthly Archives

Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives
Latest crawl: CC-MAIN-2021-31

View the Project on GitHub

Distribution of Languages

The language of a document is identified by Compact Language Detector 2 (CLD2). It is able to identify 160 different languages and up to 3 languages per document. The table lists the percentage covered by the primary language of a document (returned first by CLD2). So far, only HTML pages are passed to the language detector. The underlying data including page counts is provided in languages.csv.

crawl CC-MAIN-2021-21 CC-MAIN-2021-25 CC-MAIN-2021-31
language
<unknown> 2.8840 2.3539 1.7425
aar 0.0001 0.0001 0.0001
abk 0.0003 0.0003 0.0004
afr 0.0124 0.0130 0.0119
aka 0.0000 0.0001 0.0001
amh 0.0029 0.0030 0.0036
ara 0.6168 0.6409 0.5644
asm 0.0011 0.0018 0.0017
aym 0.0001 0.0001 0.0001
aze 0.0456 0.0458 0.0389
bak 0.0015 0.0011 0.0012
bel 0.0177 0.0174 0.0157
ben 0.0883 0.0920 0.0834
bih 0.0006 0.0005 0.0006
bis 0.0001 0.0001 0.0001
bod 0.0010 0.0009 0.0009
bos 0.0431 0.0404 0.0404
bre 0.0028 0.0022 0.0025
bul 0.2779 0.2633 0.2660
cat 0.2318 0.2263 0.2275
ceb 0.0026 0.0028 0.0015
ces 1.1124 1.0004 1.1233
chr 0.0000 0.0000 0.0000
cos 0.0024 0.0023 0.0024
crs 0.0001 0.0001 0.0000
cym 0.0115 0.0103 0.0100
dan 0.4278 0.4184 0.4191
deu 5.6906 5.5856 5.9029
div 0.0017 0.0021 0.0018
dzo 0.0000 0.0001 0.0001
ell 0.5421 0.5503 0.5452
eng 44.8424 45.1857 45.5605
epo 0.0098 0.0098 0.0067
est 0.1278 0.1313 0.1331
eus 0.0306 0.0280 0.0267
fao 0.0036 0.0031 0.0033
fas 0.6425 0.6692 0.6570
fij 0.0001 0.0001 0.0001
fin 0.3941 0.3857 0.3936
fra 4.4747 4.3725 4.5090
fry 0.0034 0.0035 0.0032
gla 0.0016 0.0020 0.0017
gle 0.0063 0.0058 0.0057
glg 0.0362 0.0356 0.0344
glv 0.0005 0.0003 0.0005
got 0.0000 0.0000 0.0000
grn 0.0007 0.0006 0.0007
guj 0.0113 0.0126 0.0111
hat 0.0017 0.0017 0.0015
hau 0.0018 0.0020 0.0024
haw 0.0004 0.0008 0.0006
heb 0.2520 0.2375 0.2467
hin 0.1291 0.1455 0.1318
hmn 0.0004 0.0008 0.0007
hrv 0.1894 0.1851 0.1860
hun 0.5655 0.5322 0.5343
hye 0.0308 0.0328 0.0276
ibo 0.0004 0.0007 0.0005
iku 0.0002 0.0002 0.0002
ile 0.0004 0.0003 0.0003
ina 0.0006 0.0006 0.0006
ind 0.8311 0.8450 0.8500
ipk 0.0002 0.0002 0.0002
isl 0.0411 0.0403 0.0407
ita 2.4277 2.4112 2.4354
jav 0.0018 0.0020 0.0023
jpn 4.5850 4.3859 4.9273
kal 0.0016 0.0017 0.0017
kan 0.0100 0.0115 0.0103
kas 0.0000 0.0000 0.0000
kat 0.0397 0.0365 0.0328
kaz 0.0217 0.0250 0.0215
kha 0.0001 0.0002 0.0002
khm 0.0081 0.0079 0.0089
kin 0.0017 0.0019 0.0018
kir 0.0049 0.0049 0.0048
kor 0.6378 0.6697 0.5698
kur 0.0062 0.0046 0.0050
lao 0.0041 0.0031 0.0037
lat 0.1060 0.1117 0.1260
lav 0.0943 0.0914 0.0886
lif 0.0000 0.0000 0.0000
lin 0.0002 0.0003 0.0002
lit 0.1660 0.1637 0.1638
ltz 0.0036 0.0033 0.0037
lug 0.0001 0.0001 0.0001
mal 0.0211 0.0211 0.0189
mar 0.0182 0.0196 0.0174
mfe 0.0001 0.0001 0.0001
mkd 0.0291 0.0294 0.0287
mlg 0.0031 0.0030 0.0028
mlt 0.0040 0.0039 0.0034
mon 0.0149 0.0138 0.0131
mri 0.0012 0.0012 0.0013
msa 0.0795 0.0823 0.0848
mya 0.0102 0.0109 0.0106
nau 0.0001 0.0000 0.0000
nep 0.0272 0.0296 0.0273
nld 1.8082 1.7758 1.9176
nno 0.0203 0.0178 0.0169
nor 0.3550 0.3480 0.3450
nso 0.0001 0.0001 0.0001
nya 0.0004 0.0005 0.0005
oci 0.0034 0.0028 0.0028
ori 0.0046 0.0039 0.0032
orm 0.0005 0.0006 0.0006
pan 0.0051 0.0048 0.0050
pol 1.6050 1.5885 1.6471
por 2.1373 2.1200 2.2068
pus 0.0040 0.0036 0.0029
que 0.0008 0.0005 0.0007
roh 0.0011 0.0011 0.0009
ron 0.6172 0.6134 0.6041
run 0.0001 0.0001 0.0001
rus 7.2658 6.7665 7.2898
sag 0.0000 0.0000 0.0000
san 0.0021 0.0019 0.0020
sco 0.0005 0.0004 0.0004
sin 0.0070 0.0075 0.0072
slk 0.3474 0.3467 0.3658
slv 0.1453 0.1465 0.1504
smo 0.0004 0.0005 0.0006
sna 0.0005 0.0006 0.0006
snd 0.0010 0.0013 0.0012
som 0.0044 0.0052 0.0048
sot 0.0004 0.0005 0.0005
spa 4.3315 4.3107 4.4105
sqi 0.0440 0.0427 0.0372
srp 0.2137 0.2016 0.2050
ssw 0.0000 0.0000 0.0000
sun 0.0009 0.0010 0.0010
sux 0.0000 0.0000 0.0000
swa 0.0071 0.0067 0.0069
swe 0.7398 0.7236 0.7292
syr 0.0007 0.0005 0.0004
tam 0.0392 0.0410 0.0395
tat 0.0078 0.0069 0.0066
tel 0.0159 0.0183 0.0168
tgk 0.0052 0.0050 0.0049
tgl 0.0071 0.0071 0.0072
tha 0.3443 0.4743 0.4134
tir 0.0007 0.0007 0.0010
ton 0.0001 0.0000 0.0001
tsn 0.0003 0.0003 0.0002
tso 0.0001 0.0000 0.0001
tuk 0.0022 0.0024 0.0021
tur 1.0099 1.0407 1.0210
uig 0.0011 0.0010 0.0011
ukr 0.4030 0.4129 0.4017
urd 0.0256 0.0267 0.0238
uzb 0.0178 0.0162 0.0151
ven 0.0000 0.0000 0.0000
vie 0.8036 1.5225 0.9478
vol 0.0012 0.0011 0.0010
war 0.0032 0.0030 0.0021
wol 0.0002 0.0001 0.0001
xho 0.0014 0.0017 0.0013
yid 0.0015 0.0015 0.0014
yor 0.0010 0.0009 0.0010
zha 0.0001 0.0000 0.0001
zho 4.6508 5.0691 4.1251
zul 0.0016 0.0019 0.0016