Statistics of Common Crawl Monthly Archives

Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives
Latest crawl: CC-MAIN-2024-26

View the Project on GitHub

Distribution of Languages

The language of a document is identified by Compact Language Detector 2 (CLD2). It is able to identify 160 different languages and up to 3 languages per document. The table lists the percentage covered by the primary language of a document (returned first by CLD2). So far, only HTML pages are passed to the language detector. The underlying data including page counts is provided in languages.csv.

crawl CC-MAIN-2024-18 CC-MAIN-2024-22 CC-MAIN-2024-26
language
<unknown> 2.8748 2.8592 3.1360
aar 0.0001 0.0001 0.0001
abk 0.0005 0.0005 0.0005
afr 0.0090 0.0092 0.0085
aka 0.0001 0.0001 0.0001
amh 0.0032 0.0033 0.0032
ara 0.6178 0.6353 0.6568
asm 0.0022 0.0021 0.0021
aym 0.0001 0.0001 0.0001
aze 0.0617 0.0591 0.0623
bak 0.0028 0.0027 0.0027
bel 0.0185 0.0197 0.0151
ben 0.1063 0.1078 0.1153
bih 0.0007 0.0006 0.0007
bis 0.0001 0.0001 0.0001
bod 0.0017 0.0016 0.0013
bos 0.0510 0.0506 0.0506
bre 0.0027 0.0020 0.0030
bul 0.2927 0.2883 0.2901
cat 0.2036 0.2046 0.2011
ceb 0.0021 0.0021 0.0019
ces 1.0126 1.1223 1.0904
chr 0.0000 0.0000 0.0000
cos 0.0021 0.0020 0.0020
crs 0.0001 0.0001 0.0001
cym 0.0107 0.0113 0.0109
dan 0.4291 0.4692 0.4529
deu 5.2631 5.5710 5.4701
div 0.0024 0.0020 0.0022
dzo 0.0001 0.0002 0.0002
ell 0.5579 0.5513 0.5475
eng 45.5105 45.9693 45.0546
epo 0.0092 0.0084 0.0084
est 0.1298 0.1407 0.1381
eus 0.0331 0.0349 0.0350
fao 0.0040 0.0042 0.0040
fas 0.6875 0.6807 0.6430
fij 0.0001 0.0001 0.0001
fin 0.3621 0.3513 0.3498
fra 4.3078 4.2157 4.2175
fry 0.0031 0.0022 0.0029
gla 0.0013 0.0014 0.0013
gle 0.0068 0.0066 0.0067
glg 0.0361 0.0383 0.0392
glv 0.0004 0.0003 0.0004
got 0.0000 0.0000 0.0000
grn 0.0007 0.0007 0.0007
guj 0.0120 0.0117 0.0119
hat 0.0018 0.0018 0.0019
hau 0.0025 0.0025 0.0030
haw 0.0005 0.0006 0.0006
heb 0.2167 0.2254 0.2241
hin 0.1859 0.1856 0.1884
hmn 0.0005 0.0005 0.0007
hrv 0.2050 0.1958 0.1961
hun 0.5826 0.5784 0.5585
hye 0.0325 0.0319 0.0329
ibo 0.0005 0.0006 0.0006
iku 0.0002 0.0002 0.0002
ile 0.0003 0.0003 0.0004
ina 0.0005 0.0005 0.0005
ind 0.9190 0.8905 0.9675
ipk 0.0000 0.0000 0.0001
isl 0.0389 0.0371 0.0385
ita 2.6120 2.4205 2.6379
jav 0.0021 0.0020 0.0021
jpn 5.1574 4.9833 4.7772
kal 0.0008 0.0009 0.0008
kan 0.0131 0.0145 0.0152
kas 0.0000 0.0000 0.0000
kat 0.0375 0.0370 0.0391
kaz 0.0240 0.0250 0.0267
kha 0.0002 0.0003 0.0004
khm 0.0110 0.0107 0.0110
kin 0.0021 0.0023 0.0022
kir 0.0085 0.0093 0.0082
kor 0.6913 0.7008 0.7992
kur 0.0061 0.0062 0.0058
lao 0.0038 0.0038 0.0034
lat 0.1141 0.1169 0.1162
lav 0.0862 0.0846 0.0880
lif 0.0000 0.0000 0.0000
lin 0.0002 0.0002 0.0003
lit 0.1561 0.1527 0.1577
ltz 0.0038 0.0037 0.0037
lug 0.0002 0.0002 0.0002
mal 0.0221 0.0226 0.0231
mar 0.0247 0.0276 0.0263
mfe 0.0001 0.0002 0.0002
mkd 0.0380 0.0373 0.0394
mlg 0.0031 0.0035 0.0035
mlt 0.0044 0.0047 0.0043
mon 0.0140 0.0144 0.0152
mri 0.0011 0.0009 0.0011
msa 0.0746 0.0740 0.0772
mya 0.0126 0.0127 0.0133
nau 0.0000 0.0000 0.0001
nep 0.0445 0.0437 0.0490
nld 1.9101 1.8350 1.8288
nno 0.0142 0.0147 0.0131
nor 0.3159 0.3066 0.3024
nso 0.0001 0.0001 0.0001
nya 0.0005 0.0005 0.0006
oci 0.0033 0.0036 0.0031
ori 0.0051 0.0054 0.0059
orm 0.0006 0.0006 0.0006
pan 0.0059 0.0058 0.0060
pol 1.7582 1.7290 1.6771
por 1.9505 2.0407 2.0232
pus 0.0034 0.0037 0.0036
que 0.0006 0.0005 0.0005
roh 0.0010 0.0011 0.0011
ron 0.5460 0.5208 0.5164
run 0.0001 0.0001 0.0001
rus 5.9477 5.9154 5.6876
sag 0.0005 0.0005 0.0003
san 0.0035 0.0035 0.0028
sco 0.0006 0.0006 0.0010
sin 0.0083 0.0097 0.0085
slk 0.3673 0.3676 0.3705
slv 0.1575 0.1469 0.1525
smo 0.0006 0.0006 0.0006
sna 0.0006 0.0006 0.0006
snd 0.0016 0.0015 0.0017
som 0.0047 0.0046 0.0045
sot 0.0006 0.0006 0.0006
spa 4.5941 4.6177 4.5025
sqi 0.0470 0.0479 0.0483
srp 0.2206 0.2178 0.2184
ssw 0.0000 0.0000 0.0001
sun 0.0009 0.0010 0.0010
sux 0.0000 0.0000 0.0000
swa 0.0080 0.0090 0.0095
swe 0.6498 0.6337 0.6279
syr 0.0003 0.0003 0.0003
tam 0.0424 0.0410 0.0434
tat 0.0068 0.0072 0.0071
tel 0.0193 0.0208 0.0210
tgk 0.0064 0.0063 0.0064
tgl 0.0088 0.0144 0.0122
tha 0.4145 0.3988 0.4189
tir 0.0008 0.0008 0.0008
ton 0.0001 0.0001 0.0001
tsn 0.0001 0.0001 0.0001
tso 0.0000 0.0000 0.0001
tuk 0.0018 0.0019 0.0019
tur 1.0623 1.1142 1.1946
uig 0.0013 0.0011 0.0012
ukr 0.4501 0.4461 0.5364
urd 0.0312 0.0307 0.0298
uzb 0.0209 0.0203 0.0208
ven 0.0000 0.0000 0.0000
vie 0.9784 0.9609 0.9859
vol 0.0005 0.0006 0.0005
war 0.0022 0.0021 0.0021
wol 0.0001 0.0001 0.0001
xho 0.0017 0.0019 0.0020
yid 0.0017 0.0018 0.0018
yor 0.0008 0.0008 0.0008
zha 0.0001 0.0001 0.0001
zho 4.4238 4.0651 4.8848
zul 0.0020 0.0022 0.0021