Statistics of Common Crawl Monthly Archives

Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives
Latest crawl: CC-MAIN-2023-06

View the Project on GitHub

Distribution of Languages

The language of a document is identified by Compact Language Detector 2 (CLD2). It is able to identify 160 different languages and up to 3 languages per document. The table lists the percentage covered by the primary language of a document (returned first by CLD2). So far, only HTML pages are passed to the language detector. The underlying data including page counts is provided in languages.csv.

crawl CC-MAIN-2022-40 CC-MAIN-2022-49 CC-MAIN-2023-06
language
<unknown> 2.5673 2.7037 2.5188
aar 0.0001 0.0002 0.0001
abk 0.0008 0.0007 0.0006
afr 0.0084 0.0079 0.0076
aka 0.0001 0.0001 0.0001
amh 0.0037 0.0031 0.0037
ara 0.6658 0.6708 0.6756
asm 0.0025 0.0025 0.0031
aym 0.0001 0.0001 0.0001
aze 0.0414 0.0402 0.0413
bak 0.0013 0.0029 0.0027
bel 0.0162 0.0156 0.0168
ben 0.0930 0.0942 0.0963
bih 0.0006 0.0007 0.0007
bis 0.0001 0.0001 0.0001
bod 0.0017 0.0015 0.0014
bos 0.0473 0.0462 0.0477
bre 0.0029 0.0027 0.0028
bul 0.2900 0.2922 0.2970
cat 0.2314 0.2369 0.2335
ceb 0.0024 0.0022 0.0024
ces 1.1069 1.0989 1.0994
chr 0.0001 0.0001 0.0001
cos 0.0026 0.0024 0.0025
crs 0.0001 0.0001 0.0001
cym 0.0095 0.0102 0.0100
dan 0.4301 0.4473 0.4501
deu 5.8811 5.8640 5.9010
div 0.0021 0.0023 0.0023
dzo 0.0002 0.0002 0.0002
ell 0.5870 0.6068 0.6099
eng 45.8786 46.3044 46.2733
epo 0.0091 0.0091 0.0096
est 0.1354 0.1368 0.1376
eus 0.0323 0.0340 0.0332
fao 0.0040 0.0042 0.0042
fas 0.6582 0.6671 0.6589
fij 0.0002 0.0001 0.0001
fin 0.4080 0.4165 0.4156
fra 4.7254 4.7768 4.7487
fry 0.0032 0.0031 0.0029
gla 0.0015 0.0016 0.0014
gle 0.0063 0.0065 0.0066
glg 0.0400 0.0408 0.0425
glv 0.0005 0.0004 0.0004
got 0.0000 0.0000 0.0000
grn 0.0006 0.0006 0.0007
guj 0.0126 0.0125 0.0128
hat 0.0018 0.0018 0.0018
hau 0.0028 0.0026 0.0031
haw 0.0007 0.0007 0.0006
heb 0.1152 0.1170 0.1150
hin 0.1588 0.1592 0.1747
hmn 0.0006 0.0007 0.0007
hrv 0.1979 0.1990 0.2035
hun 0.6093 0.6203 0.6018
hye 0.0314 0.0315 0.0324
ibo 0.0006 0.0006 0.0007
iku 0.0002 0.0002 0.0002
ile 0.0003 0.0003 0.0003
ina 0.0007 0.0007 0.0007
ind 0.7991 0.7992 0.8132
ipk 0.0001 0.0001 0.0001
isl 0.0397 0.0404 0.0418
ita 2.5712 2.6835 2.6843
jav 0.0023 0.0023 0.0025
jpn 4.7884 4.4586 4.7648
kal 0.0010 0.0010 0.0010
kan 0.0122 0.0121 0.0135
kas 0.0000 0.0000 0.0000
kat 0.0352 0.0369 0.0358
kaz 0.0211 0.0212 0.0205
kha 0.0002 0.0002 0.0002
khm 0.0108 0.0106 0.0131
kin 0.0020 0.0019 0.0020
kir 0.0049 0.0052 0.0053
kor 0.6498 0.6526 0.6518
kur 0.0055 0.0056 0.0054
lao 0.0045 0.0041 0.0044
lat 0.1232 0.1059 0.1158
lav 0.0831 0.0832 0.0848
lif 0.0000 0.0000 0.0000
lin 0.0002 0.0002 0.0002
lit 0.1617 0.1609 0.1626
ltz 0.0037 0.0038 0.0040
lug 0.0002 0.0002 0.0002
mal 0.0222 0.0219 0.0227
mar 0.0213 0.0212 0.0223
mfe 0.0002 0.0002 0.0002
mkd 0.0347 0.0364 0.0360
mlg 0.0031 0.0027 0.0030
mlt 0.0042 0.0043 0.0041
mon 0.0216 0.0238 0.0169
mri 0.0013 0.0013 0.0012
msa 0.0826 0.0812 0.0812
mya 0.0126 0.0118 0.0125
nau 0.0001 0.0000 0.0000
nep 0.0304 0.0306 0.0327
nld 2.0585 2.1504 2.0693
nno 0.0162 0.0159 0.0154
nor 0.3421 0.3454 0.3401
nso 0.0001 0.0001 0.0001
nya 0.0006 0.0006 0.0006
oci 0.0032 0.0033 0.0031
ori 0.0044 0.0042 0.0049
orm 0.0005 0.0004 0.0006
pan 0.0061 0.0062 0.0063
pol 1.6636 1.6881 1.6977
por 1.1505 1.1535 1.1660
pus 0.0028 0.0029 0.0030
que 0.0007 0.0007 0.0007
roh 0.0012 0.0012 0.0012
ron 0.5637 0.5772 0.5727
run 0.0001 0.0001 0.0001
rus 5.9692 5.6647 5.7910
sag 0.0002 0.0001 0.0002
san 0.0029 0.0028 0.0032
sco 0.0007 0.0006 0.0007
sin 0.0085 0.0085 0.0078
slk 0.3777 0.3890 0.3971
slv 0.1582 0.1625 0.1600
smo 0.0006 0.0005 0.0006
sna 0.0007 0.0006 0.0006
snd 0.0015 0.0014 0.0016
som 0.0040 0.0039 0.0041
sot 0.0007 0.0006 0.0006
spa 4.4690 4.5435 4.5977
sqi 0.0437 0.0450 0.0463
srp 0.2205 0.2228 0.2236
ssw 0.0000 0.0001 0.0000
sun 0.0012 0.0011 0.0014
sux 0.0000 0.0000 0.0000
swa 0.0077 0.0075 0.0077
swe 0.6969 0.6899 0.7016
syr 0.0005 0.0003 0.0003
tam 0.0446 0.0450 0.0463
tat 0.0063 0.0078 0.0078
tel 0.0183 0.0181 0.0189
tgk 0.0047 0.0049 0.0050
tgl 0.0088 0.0086 0.0073
tha 0.4143 0.4027 0.4051
tir 0.0009 0.0008 0.0010
ton 0.0001 0.0001 0.0001
tsn 0.0001 0.0001 0.0001
tso 0.0000 0.0000 0.0000
tuk 0.0027 0.0026 0.0027
tur 0.8439 0.8040 0.8221
uig 0.0015 0.0013 0.0014
ukr 0.3304 0.3350 0.3358
urd 0.0274 0.0277 0.0298
uzb 0.0162 0.0150 0.0160
ven 0.0000 0.0000 0.0000
vie 1.0299 0.9937 1.0119
vol 0.0006 0.0005 0.0005
war 0.0027 0.0026 0.0024
wol 0.0001 0.0001 0.0001
xho 0.0015 0.0015 0.0011
yid 0.0017 0.0015 0.0016
yor 0.0010 0.0010 0.0010
zha 0.0001 0.0001 0.0001
zho 4.8747 4.5969 4.2807
zul 0.0017 0.0017 0.0014