Statistics of Common Crawl Monthly Archives

Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives
Latest crawl: CC-MAIN-2023-23

View the Project on GitHub

Distribution of Languages

The language of a document is identified by Compact Language Detector 2 (CLD2). It is able to identify 160 different languages and up to 3 languages per document. The table lists the percentage covered by the primary language of a document (returned first by CLD2). So far, only HTML pages are passed to the language detector. The underlying data including page counts is provided in languages.csv.

crawl CC-MAIN-2023-06 CC-MAIN-2023-14 CC-MAIN-2023-23
language
<unknown> 2.5188 2.8514 2.8762
aar 0.0001 0.0001 0.0002
abk 0.0006 0.0005 0.0005
afr 0.0076 0.0073 0.0070
aka 0.0001 0.0001 0.0001
amh 0.0037 0.0036 0.0037
ara 0.6756 0.6790 0.6381
asm 0.0031 0.0026 0.0025
aym 0.0001 0.0001 0.0001
aze 0.0413 0.0446 0.0440
bak 0.0027 0.0026 0.0027
bel 0.0168 0.0166 0.0151
ben 0.0963 0.1036 0.0965
bih 0.0007 0.0008 0.0007
bis 0.0001 0.0001 0.0001
bod 0.0014 0.0014 0.0013
bos 0.0477 0.0462 0.0461
bre 0.0028 0.0025 0.0028
bul 0.2970 0.2961 0.3062
cat 0.2335 0.2363 0.2433
ceb 0.0024 0.0023 0.0016
ces 1.0994 1.1069 1.1482
chr 0.0001 0.0000 0.0001
cos 0.0025 0.0025 0.0024
crs 0.0001 0.0001 0.0001
cym 0.0100 0.0101 0.0094
dan 0.4501 0.4384 0.4287
deu 5.9010 5.7651 6.0319
div 0.0023 0.0024 0.0023
dzo 0.0002 0.0001 0.0001
ell 0.6099 0.6088 0.5938
eng 46.2733 46.1750 45.2170
epo 0.0096 0.0096 0.0090
est 0.1376 0.1387 0.1410
eus 0.0332 0.0350 0.0346
fao 0.0042 0.0042 0.0041
fas 0.6589 0.6002 0.7105
fij 0.0001 0.0001 0.0001
fin 0.4156 0.3965 0.4007
fra 4.7487 4.6770 4.7691
fry 0.0029 0.0028 0.0029
gla 0.0014 0.0016 0.0015
gle 0.0066 0.0070 0.0065
glg 0.0425 0.0418 0.0417
glv 0.0004 0.0003 0.0004
got 0.0000 0.0000 0.0000
grn 0.0007 0.0006 0.0006
guj 0.0128 0.0142 0.0142
hat 0.0018 0.0020 0.0019
hau 0.0031 0.0036 0.0035
haw 0.0006 0.0008 0.0006
heb 0.1150 0.1140 0.1110
hin 0.1747 0.1790 0.1770
hmn 0.0007 0.0008 0.0007
hrv 0.2035 0.1882 0.1857
hun 0.6018 0.6120 0.6161
hye 0.0324 0.0337 0.0307
ibo 0.0007 0.0008 0.0006
iku 0.0002 0.0002 0.0002
ile 0.0003 0.0003 0.0003
ina 0.0007 0.0006 0.0005
ind 0.8132 0.7898 0.7676
ipk 0.0001 0.0001 0.0001
isl 0.0418 0.0400 0.0412
ita 2.6843 2.7261 2.7524
jav 0.0025 0.0046 0.0023
jpn 4.7648 4.4368 5.0335
kal 0.0010 0.0007 0.0009
kan 0.0135 0.0134 0.0132
kas 0.0000 0.0000 0.0000
kat 0.0358 0.0383 0.0369
kaz 0.0205 0.0224 0.0214
kha 0.0002 0.0003 0.0003
khm 0.0131 0.0245 0.0110
kin 0.0020 0.0021 0.0018
kir 0.0053 0.0051 0.0051
kor 0.6518 0.6576 0.6760
kur 0.0054 0.0062 0.0059
lao 0.0044 0.0043 0.0039
lat 0.1158 0.1099 0.1179
lav 0.0848 0.0855 0.0864
lif 0.0000 0.0000 0.0000
lin 0.0002 0.0002 0.0002
lit 0.1626 0.1668 0.1683
ltz 0.0040 0.0041 0.0043
lug 0.0002 0.0002 0.0002
mal 0.0227 0.0231 0.0229
mar 0.0223 0.0249 0.0237
mfe 0.0002 0.0001 0.0001
mkd 0.0360 0.0356 0.0341
mlg 0.0030 0.0026 0.0029
mlt 0.0041 0.0042 0.0043
mon 0.0169 0.0165 0.0161
mri 0.0012 0.0013 0.0011
msa 0.0812 0.0774 0.0785
mya 0.0125 0.0127 0.0116
nau 0.0000 0.0000 0.0000
nep 0.0327 0.0327 0.0327
nld 2.0693 2.1208 2.1337
nno 0.0154 0.0165 0.0155
nor 0.3401 0.3462 0.3404
nso 0.0001 0.0000 0.0001
nya 0.0006 0.0008 0.0006
oci 0.0031 0.0033 0.0033
ori 0.0049 0.0053 0.0052
orm 0.0006 0.0005 0.0005
pan 0.0063 0.0069 0.0062
pol 1.6977 1.6429 1.6840
por 1.1660 1.1307 1.1399
pus 0.0030 0.0033 0.0032
que 0.0007 0.0007 0.0007
roh 0.0012 0.0013 0.0014
ron 0.5727 0.5272 0.4999
run 0.0001 0.0001 0.0001
rus 5.7910 5.8227 5.9898
sag 0.0002 0.0001 0.0001
san 0.0032 0.0032 0.0032
sco 0.0007 0.0006 0.0007
sin 0.0078 0.0081 0.0080
slk 0.3971 0.3984 0.4032
slv 0.1600 0.1627 0.1656
smo 0.0006 0.0006 0.0005
sna 0.0006 0.0007 0.0006
snd 0.0016 0.0017 0.0014
som 0.0041 0.0042 0.0039
sot 0.0006 0.0007 0.0006
spa 4.5977 4.5480 4.6013
sqi 0.0463 0.0475 0.0435
srp 0.2236 0.2160 0.2182
ssw 0.0000 0.0000 0.0000
sun 0.0014 0.0011 0.0011
sux 0.0000 0.0000 0.0000
swa 0.0077 0.0080 0.0084
swe 0.7016 0.7001 0.7215
syr 0.0003 0.0003 0.0003
tam 0.0463 0.0452 0.0460
tat 0.0078 0.0083 0.0064
tel 0.0189 0.0194 0.0184
tgk 0.0050 0.0051 0.0054
tgl 0.0073 0.0072 0.0072
tha 0.4051 0.4402 0.4061
tir 0.0010 0.0008 0.0009
ton 0.0001 0.0001 0.0001
tsn 0.0001 0.0001 0.0001
tso 0.0000 0.0000 0.0000
tuk 0.0027 0.0022 0.0020
tur 0.8221 0.7897 0.7667
uig 0.0014 0.0015 0.0013
ukr 0.3358 0.3633 0.3813
urd 0.0298 0.0291 0.0273
uzb 0.0160 0.0173 0.0168
ven 0.0000 0.0000 0.0000
vie 1.0119 1.0555 1.0275
vol 0.0005 0.0005 0.0006
war 0.0024 0.0026 0.0019
wol 0.0001 0.0001 0.0001
xho 0.0011 0.0011 0.0019
yid 0.0016 0.0018 0.0016
yor 0.0010 0.0011 0.0010
zha 0.0001 0.0001 0.0001
zho 4.2807 4.6316 4.3037
zul 0.0014 0.0016 0.0020