Statistics of Common Crawl Monthly Archives

Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives
Latest crawl: CC-MAIN-2021-39

View the Project on GitHub

Distribution of Languages

The language of a document is identified by Compact Language Detector 2 (CLD2). It is able to identify 160 different languages and up to 3 languages per document. The table lists the percentage covered by the primary language of a document (returned first by CLD2). So far, only HTML pages are passed to the language detector. The underlying data including page counts is provided in languages.csv.

crawl CC-MAIN-2021-25 CC-MAIN-2021-31 CC-MAIN-2021-39
language
<unknown> 2.3539 1.7425 2.8379
aar 0.0001 0.0001 0.0002
abk 0.0003 0.0004 0.0004
afr 0.0130 0.0119 0.0133
aka 0.0001 0.0001 0.0001
amh 0.0030 0.0036 0.0034
ara 0.6409 0.5644 0.5981
asm 0.0018 0.0017 0.0017
aym 0.0001 0.0001 0.0001
aze 0.0458 0.0389 0.0445
bak 0.0011 0.0012 0.0011
bel 0.0174 0.0157 0.0179
ben 0.0920 0.0834 0.0890
bih 0.0005 0.0006 0.0006
bis 0.0001 0.0001 0.0001
bod 0.0009 0.0009 0.0011
bos 0.0404 0.0404 0.0428
bre 0.0022 0.0025 0.0024
bul 0.2633 0.2660 0.2714
cat 0.2263 0.2275 0.2360
ceb 0.0028 0.0015 0.0023
ces 1.0004 1.1233 1.0185
chr 0.0000 0.0000 0.0000
cos 0.0023 0.0024 0.0026
crs 0.0001 0.0000 0.0001
cym 0.0103 0.0100 0.0103
dan 0.4184 0.4191 0.4049
deu 5.5856 5.9029 5.6073
div 0.0021 0.0018 0.0020
dzo 0.0001 0.0001 0.0001
ell 0.5503 0.5452 0.5440
eng 45.1857 45.5605 45.4257
epo 0.0098 0.0067 0.0090
est 0.1313 0.1331 0.1336
eus 0.0280 0.0267 0.0278
fao 0.0031 0.0033 0.0034
fas 0.6692 0.6570 0.6371
fij 0.0001 0.0001 0.0001
fin 0.3857 0.3936 0.3904
fra 4.3725 4.5090 4.4436
fry 0.0035 0.0032 0.0037
gla 0.0020 0.0017 0.0021
gle 0.0058 0.0057 0.0059
glg 0.0356 0.0344 0.0379
glv 0.0003 0.0005 0.0005
got 0.0000 0.0000 0.0000
grn 0.0006 0.0007 0.0007
guj 0.0126 0.0111 0.0118
hat 0.0017 0.0015 0.0015
hau 0.0020 0.0024 0.0020
haw 0.0008 0.0006 0.0007
heb 0.2375 0.2467 0.2431
hin 0.1455 0.1318 0.1340
hmn 0.0008 0.0007 0.0007
hrv 0.1851 0.1860 0.1939
hun 0.5322 0.5343 0.5384
hye 0.0328 0.0276 0.0315
ibo 0.0007 0.0005 0.0006
iku 0.0002 0.0002 0.0002
ile 0.0003 0.0003 0.0003
ina 0.0006 0.0006 0.0005
ind 0.8450 0.8500 0.8609
ipk 0.0002 0.0002 0.0001
isl 0.0403 0.0407 0.0412
ita 2.4112 2.4354 2.3867
jav 0.0020 0.0023 0.0022
jpn 4.3859 4.9273 4.5011
kal 0.0017 0.0017 0.0017
kan 0.0115 0.0103 0.0110
kas 0.0000 0.0000 0.0000
kat 0.0365 0.0328 0.0355
kaz 0.0250 0.0215 0.0246
kha 0.0002 0.0002 0.0002
khm 0.0079 0.0089 0.0077
kin 0.0019 0.0018 0.0020
kir 0.0049 0.0048 0.0049
kor 0.6697 0.5698 0.6851
kur 0.0046 0.0050 0.0049
lao 0.0031 0.0037 0.0038
lat 0.1117 0.1260 0.1095
lav 0.0914 0.0886 0.0920
lif 0.0000 0.0000 0.0000
lin 0.0003 0.0002 0.0003
lit 0.1637 0.1638 0.1714
ltz 0.0033 0.0037 0.0036
lug 0.0001 0.0001 0.0001
mal 0.0211 0.0189 0.0213
mar 0.0196 0.0174 0.0184
mfe 0.0001 0.0001 0.0001
mkd 0.0294 0.0287 0.0313
mlg 0.0030 0.0028 0.0027
mlt 0.0039 0.0034 0.0043
mon 0.0138 0.0131 0.0156
mri 0.0012 0.0013 0.0016
msa 0.0823 0.0848 0.0831
mya 0.0109 0.0106 0.0110
nau 0.0000 0.0000 0.0000
nep 0.0296 0.0273 0.0319
nld 1.7758 1.9176 1.7718
nno 0.0178 0.0169 0.0186
nor 0.3480 0.3450 0.3381
nso 0.0001 0.0001 0.0001
nya 0.0005 0.0005 0.0006
oci 0.0028 0.0028 0.0031
ori 0.0039 0.0032 0.0034
orm 0.0006 0.0006 0.0005
pan 0.0048 0.0050 0.0057
pol 1.5885 1.6471 1.6640
por 2.1200 2.2068 2.0955
pus 0.0036 0.0029 0.0031
que 0.0005 0.0007 0.0006
roh 0.0011 0.0009 0.0010
ron 0.6134 0.6041 0.6122
run 0.0001 0.0001 0.0001
rus 6.7665 7.2898 6.5317
sag 0.0000 0.0000 0.0000
san 0.0019 0.0020 0.0021
sco 0.0004 0.0004 0.0004
sin 0.0075 0.0072 0.0075
slk 0.3467 0.3658 0.3567
slv 0.1465 0.1504 0.1534
smo 0.0005 0.0006 0.0005
sna 0.0006 0.0006 0.0007
snd 0.0013 0.0012 0.0012
som 0.0052 0.0048 0.0049
sot 0.0005 0.0005 0.0005
spa 4.3107 4.4105 4.3514
sqi 0.0427 0.0372 0.0396
srp 0.2016 0.2050 0.2120
ssw 0.0000 0.0000 0.0000
sun 0.0010 0.0010 0.0010
sux 0.0000 0.0000 0.0000
swa 0.0067 0.0069 0.0076
swe 0.7236 0.7292 0.6985
syr 0.0005 0.0004 0.0005
tam 0.0410 0.0395 0.0427
tat 0.0069 0.0066 0.0083
tel 0.0183 0.0168 0.0189
tgk 0.0050 0.0049 0.0053
tgl 0.0071 0.0072 0.0076
tha 0.4743 0.4134 0.4804
tir 0.0007 0.0010 0.0007
ton 0.0000 0.0001 0.0000
tsn 0.0003 0.0002 0.0001
tso 0.0000 0.0001 0.0000
tuk 0.0024 0.0021 0.0024
tur 1.0407 1.0210 0.9991
uig 0.0010 0.0011 0.0009
ukr 0.4129 0.4017 0.4127
urd 0.0267 0.0238 0.0280
uzb 0.0162 0.0151 0.0161
ven 0.0000 0.0000 0.0000
vie 1.5225 0.9478 0.8826
vol 0.0011 0.0010 0.0008
war 0.0030 0.0021 0.0029
wol 0.0001 0.0001 0.0002
xho 0.0017 0.0013 0.0020
yid 0.0015 0.0014 0.0015
yor 0.0009 0.0010 0.0010
zha 0.0000 0.0001 0.0000
zho 5.0691 4.1251 4.9885
zul 0.0019 0.0016 0.0020