Statistics of Common Crawl Monthly Archives

Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives
Latest crawl: CC-MAIN-2024-18

View the Project on GitHub

Distribution of Languages

The language of a document is identified by Compact Language Detector 2 (CLD2). It is able to identify 160 different languages and up to 3 languages per document. The table lists the percentage covered by the primary language of a document (returned first by CLD2). So far, only HTML pages are passed to the language detector. The underlying data including page counts is provided in languages.csv.

crawl CC-MAIN-2023-50 CC-MAIN-2024-10 CC-MAIN-2024-18
language
<unknown> 3.2338 2.7619 2.8748
aar 0.0001 0.0001 0.0001
abk 0.0005 0.0004 0.0005
afr 0.0091 0.0081 0.0090
aka 0.0001 0.0001 0.0001
amh 0.0033 0.0031 0.0032
ara 0.5867 0.5875 0.6178
asm 0.0020 0.0020 0.0022
aym 0.0000 0.0001 0.0001
aze 0.0528 0.0603 0.0617
bak 0.0028 0.0025 0.0028
bel 0.0149 0.0158 0.0185
ben 0.0981 0.1025 0.1063
bih 0.0007 0.0006 0.0007
bis 0.0001 0.0001 0.0001
bod 0.0015 0.0018 0.0017
bos 0.0475 0.0480 0.0510
bre 0.0026 0.0017 0.0027
bul 0.2966 0.2877 0.2927
cat 0.2096 0.2053 0.2036
ceb 0.0015 0.0017 0.0021
ces 1.0206 0.9899 1.0126
chr 0.0000 0.0000 0.0000
cos 0.0023 0.0019 0.0021
crs 0.0001 0.0001 0.0001
cym 0.0100 0.0103 0.0107
dan 0.4306 0.4094 0.4291
deu 5.4499 5.3948 5.2631
div 0.0025 0.0026 0.0024
dzo 0.0001 0.0001 0.0001
ell 0.5806 0.5399 0.5579
eng 44.4285 46.4536 45.5105
epo 0.0090 0.0080 0.0092
est 0.1339 0.1290 0.1298
eus 0.0328 0.0329 0.0331
fao 0.0043 0.0041 0.0040
fas 0.6795 0.6666 0.6875
fij 0.0001 0.0001 0.0001
fin 0.3680 0.3577 0.3621
fra 4.3933 4.3263 4.3078
fry 0.0031 0.0024 0.0031
gla 0.0013 0.0013 0.0013
gle 0.0059 0.0061 0.0068
glg 0.0351 0.0335 0.0361
glv 0.0003 0.0003 0.0004
got 0.0000 0.0000 0.0000
grn 0.0006 0.0006 0.0007
guj 0.0114 0.0113 0.0120
hat 0.0017 0.0016 0.0018
hau 0.0021 0.0021 0.0025
haw 0.0006 0.0005 0.0005
heb 0.2091 0.2205 0.2167
hin 0.1749 0.1784 0.1859
hmn 0.0006 0.0005 0.0005
hrv 0.2014 0.1961 0.2050
hun 0.6695 0.5646 0.5826
hye 0.0300 0.0298 0.0325
ibo 0.0005 0.0006 0.0005
iku 0.0002 0.0002 0.0002
ile 0.0003 0.0003 0.0003
ina 0.0005 0.0005 0.0005
ind 0.8596 0.8645 0.9190
ipk 0.0001 0.0000 0.0000
isl 0.0392 0.0399 0.0389
ita 2.5834 2.5737 2.6120
jav 0.0023 0.0020 0.0021
jpn 5.1508 5.0934 5.1574
kal 0.0008 0.0009 0.0008
kan 0.0123 0.0128 0.0131
kas 0.0000 0.0000 0.0000
kat 0.0360 0.0362 0.0375
kaz 0.0221 0.0220 0.0240
kha 0.0003 0.0003 0.0002
khm 0.0097 0.0101 0.0110
kin 0.0019 0.0020 0.0021
kir 0.0056 0.0077 0.0085
kor 0.7288 0.6504 0.6913
kur 0.0056 0.0063 0.0061
lao 0.0036 0.0034 0.0038
lat 0.1169 0.1127 0.1141
lav 0.0849 0.0836 0.0862
lif 0.0000 0.0000 0.0000
lin 0.0002 0.0002 0.0002
lit 0.1553 0.1531 0.1561
ltz 0.0039 0.0034 0.0038
lug 0.0002 0.0002 0.0002
mal 0.0218 0.0215 0.0221
mar 0.0212 0.0234 0.0247
mfe 0.0002 0.0002 0.0001
mkd 0.0334 0.0357 0.0380
mlg 0.0032 0.0028 0.0031
mlt 0.0042 0.0039 0.0044
mon 0.0142 0.0141 0.0140
mri 0.0010 0.0009 0.0011
msa 0.0742 0.0749 0.0746
mya 0.0118 0.0113 0.0126
nau 0.0000 0.0000 0.0000
nep 0.0395 0.0428 0.0445
nld 1.9746 1.9608 1.9101
nno 0.0137 0.0138 0.0142
nor 0.3191 0.3197 0.3159
nso 0.0001 0.0001 0.0001
nya 0.0005 0.0005 0.0005
oci 0.0031 0.0027 0.0033
ori 0.0044 0.0048 0.0051
orm 0.0006 0.0005 0.0006
pan 0.0055 0.0056 0.0059
pol 1.7490 1.7245 1.7582
por 1.7631 1.9278 1.9505
pus 0.0028 0.0029 0.0034
que 0.0006 0.0006 0.0006
roh 0.0013 0.0013 0.0010
ron 0.5690 0.5278 0.5460
run 0.0001 0.0001 0.0001
rus 6.0303 5.8095 5.9477
sag 0.0004 0.0001 0.0005
san 0.0031 0.0030 0.0035
sco 0.0007 0.0006 0.0006
sin 0.0077 0.0080 0.0083
slk 0.3590 0.3608 0.3673
slv 0.1631 0.1510 0.1575
smo 0.0005 0.0005 0.0006
sna 0.0005 0.0005 0.0006
snd 0.0015 0.0016 0.0016
som 0.0041 0.0043 0.0047
sot 0.0005 0.0005 0.0006
spa 4.5391 4.5462 4.5941
sqi 0.0439 0.0450 0.0470
srp 0.2206 0.2150 0.2206
ssw 0.0000 0.0000 0.0000
sun 0.0016 0.0011 0.0009
sux 0.0000 0.0000 0.0000
swa 0.0073 0.0075 0.0080
swe 0.6637 0.6494 0.6498
syr 0.0003 0.0003 0.0003
tam 0.0423 0.0425 0.0424
tat 0.0062 0.0064 0.0068
tel 0.0161 0.0169 0.0193
tgk 0.0052 0.0051 0.0064
tgl 0.0093 0.0084 0.0088
tha 0.4251 0.4101 0.4145
tir 0.0008 0.0008 0.0008
ton 0.0001 0.0001 0.0001
tsn 0.0001 0.0001 0.0001
tso 0.0000 0.0000 0.0000
tuk 0.0025 0.0018 0.0018
tur 0.9804 0.9864 1.0623
uig 0.0013 0.0012 0.0013
ukr 0.3845 0.4295 0.4501
urd 0.0265 0.0284 0.0312
uzb 0.0198 0.0220 0.0209
ven 0.0000 0.0000 0.0000
vie 1.0291 0.9873 0.9784
vol 0.0005 0.0004 0.0005
war 0.0015 0.0019 0.0022
wol 0.0002 0.0001 0.0001
xho 0.0017 0.0017 0.0017
yid 0.0016 0.0015 0.0017
yor 0.0008 0.0008 0.0008
zha 0.0000 0.0000 0.0001
zho 5.0798 4.1666 4.4238
zul 0.0018 0.0018 0.0020