Statistics of Common Crawl Monthly Archives

Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives
Latest crawl: CC-MAIN-2022-05

View the Project on GitHub

Distribution of Languages

The language of a document is identified by Compact Language Detector 2 (CLD2). It is able to identify 160 different languages and up to 3 languages per document. The table lists the percentage covered by the primary language of a document (returned first by CLD2). So far, only HTML pages are passed to the language detector. The underlying data including page counts is provided in languages.csv.

crawl CC-MAIN-2021-43 CC-MAIN-2021-49 CC-MAIN-2022-05
language
<unknown> 2.7811 3.7199 2.7625
aar 0.0001 0.0002 0.0002
abk 0.0004 0.0004 0.0004
afr 0.0116 0.0127 0.0143
aka 0.0001 0.0001 0.0001
amh 0.0033 0.0027 0.0031
ara 0.5796 0.6469 0.6082
asm 0.0016 0.0016 0.0016
aym 0.0001 0.0000 0.0000
aze 0.0399 0.0451 0.0405
bak 0.0012 0.0010 0.0020
bel 0.0151 0.0158 0.0153
ben 0.0843 0.0857 0.0806
bih 0.0006 0.0004 0.0006
bis 0.0001 0.0001 0.0001
bod 0.0009 0.0010 0.0010
bos 0.0411 0.0439 0.0415
bre 0.0024 0.0026 0.0024
bul 0.2594 0.2556 0.2631
cat 0.2333 0.2332 0.2341
ceb 0.0021 0.0023 0.0020
ces 1.0096 0.9165 1.0039
chr 0.0000 0.0000 0.0000
cos 0.0025 0.0024 0.0024
crs 0.0001 0.0001 0.0001
cym 0.0112 0.0135 0.0120
dan 0.4135 0.4006 0.4180
deu 5.6813 5.4084 5.6032
div 0.0018 0.0020 0.0021
dzo 0.0001 0.0001 0.0001
ell 0.5428 0.5465 0.5396
eng 45.3925 46.2521 46.4560
epo 0.0088 0.0100 0.0086
est 0.1360 0.1354 0.1353
eus 0.0290 0.0326 0.0313
fao 0.0035 0.0033 0.0037
fas 0.6293 0.5881 0.6130
fij 0.0001 0.0001 0.0002
fin 0.3961 0.3825 0.3947
fra 4.4173 4.3941 4.4552
fry 0.0031 0.0033 0.0034
gla 0.0020 0.0028 0.0018
gle 0.0055 0.0058 0.0057
glg 0.0376 0.0408 0.0402
glv 0.0004 0.0003 0.0004
got 0.0000 0.0000 0.0000
grn 0.0006 0.0005 0.0007
guj 0.0103 0.0114 0.0111
hat 0.0014 0.0016 0.0013
hau 0.0033 0.0032 0.0027
haw 0.0005 0.0006 0.0007
heb 0.2388 0.2399 0.2347
hin 0.1268 0.1325 0.1279
hmn 0.0005 0.0007 0.0006
hrv 0.1910 0.1970 0.1950
hun 0.5446 0.5428 0.5480
hye 0.0302 0.0328 0.0300
ibo 0.0005 0.0006 0.0006
iku 0.0002 0.0002 0.0002
ile 0.0003 0.0003 0.0004
ina 0.0005 0.0006 0.0006
ind 0.8331 0.7630 0.7605
ipk 0.0001 0.0001 0.0001
isl 0.0377 0.0365 0.0378
ita 2.4099 2.3929 2.3877
jav 0.0020 0.0019 0.0019
jpn 4.7534 4.2860 4.7738
kal 0.0012 0.0011 0.0010
kan 0.0095 0.0108 0.0103
kas 0.0000 0.0000 0.0000
kat 0.0357 0.0393 0.0362
kaz 0.0223 0.0220 0.0212
kha 0.0002 0.0002 0.0002
khm 0.0091 0.0085 0.0083
kin 0.0020 0.0020 0.0017
kir 0.0051 0.0054 0.0049
kor 0.6436 0.7362 0.6723
kur 0.0051 0.0057 0.0056
lao 0.0039 0.0037 0.0037
lat 0.1154 0.0726 0.0996
lav 0.0888 0.0911 0.0825
lif 0.0000 0.0000 0.0000
lin 0.0002 0.0002 0.0002
lit 0.1696 0.1750 0.1568
ltz 0.0037 0.0033 0.0033
lug 0.0001 0.0001 0.0001
mal 0.0185 0.0193 0.0192
mar 0.0177 0.0192 0.0185
mfe 0.0001 0.0001 0.0001
mkd 0.0298 0.0321 0.0305
mlg 0.0025 0.0023 0.0023
mlt 0.0042 0.0054 0.0042
mon 0.0148 0.0172 0.0160
mri 0.0014 0.0015 0.0012
msa 0.0801 0.0731 0.0768
mya 0.0112 0.0109 0.0109
nau 0.0000 0.0000 0.0000
nep 0.0287 0.0271 0.0273
nld 1.8081 1.6558 1.7735
nno 0.0169 0.0160 0.0161
nor 0.3379 0.3235 0.3188
nso 0.0001 0.0001 0.0001
nya 0.0004 0.0007 0.0005
oci 0.0027 0.0027 0.0032
ori 0.0039 0.0048 0.0035
orm 0.0004 0.0004 0.0004
pan 0.0047 0.0050 0.0051
pol 1.6216 1.5780 1.6081
por 2.1815 2.1088 2.0693
pus 0.0030 0.0035 0.0028
que 0.0007 0.0005 0.0006
roh 0.0013 0.0012 0.0009
ron 0.6038 0.5988 0.5835
run 0.0001 0.0001 0.0001
rus 6.7970 5.9948 6.2413
sag 0.0000 0.0000 0.0000
san 0.0027 0.0025 0.0021
sco 0.0005 0.0005 0.0005
sin 0.0080 0.0083 0.0077
slk 0.3463 0.3320 0.3399
slv 0.1477 0.1486 0.1586
smo 0.0004 0.0006 0.0005
sna 0.0006 0.0006 0.0005
snd 0.0010 0.0012 0.0010
som 0.0047 0.0042 0.0042
sot 0.0004 0.0006 0.0004
spa 4.3148 4.1590 4.2539
sqi 0.0373 0.0403 0.0368
srp 0.2075 0.2052 0.2049
ssw 0.0000 0.0000 0.0000
sun 0.0010 0.0011 0.0010
sux 0.0000 0.0000 0.0000
swa 0.0072 0.0081 0.0075
swe 0.7049 0.6645 0.6822
syr 0.0006 0.0008 0.0006
tam 0.0399 0.0423 0.0402
tat 0.0075 0.0073 0.0080
tel 0.0154 0.0169 0.0165
tgk 0.0041 0.0040 0.0042
tgl 0.0077 0.0087 0.0083
tha 0.3695 0.3884 0.3781
tir 0.0009 0.0005 0.0007
ton 0.0001 0.0001 0.0001
tsn 0.0001 0.0001 0.0001
tso 0.0001 0.0001 0.0001
tuk 0.0022 0.0026 0.0021
tur 0.9632 0.8845 0.8550
uig 0.0011 0.0011 0.0010
ukr 0.4030 0.3865 0.4023
urd 0.0259 0.0272 0.0257
uzb 0.0154 0.0156 0.0148
ven 0.0000 0.0000 0.0000
vie 0.8428 0.7869 0.8379
vol 0.0008 0.0009 0.0009
war 0.0025 0.0029 0.0027
wol 0.0002 0.0001 0.0002
xho 0.0021 0.0029 0.0022
yid 0.0015 0.0018 0.0016
yor 0.0010 0.0011 0.0011
zha 0.0001 0.0000 0.0001
zho 4.8286 5.2986 4.8282
zul 0.0020 0.0029 0.0024