Statistics of Common Crawl Monthly Archives

Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives
Latest crawl: CC-MAIN-2022-21

View the Project on GitHub

Distribution of Languages

The language of a document is identified by Compact Language Detector 2 (CLD2). It is able to identify 160 different languages and up to 3 languages per document. The table lists the percentage covered by the primary language of a document (returned first by CLD2). So far, only HTML pages are passed to the language detector. The underlying data including page counts is provided in languages.csv.

crawl CC-MAIN-2021-49 CC-MAIN-2022-05 CC-MAIN-2022-21
language
<unknown> 3.7199 2.7625 2.5879
aar 0.0002 0.0002 0.0002
abk 0.0004 0.0004 0.0005
afr 0.0127 0.0143 0.0108
aka 0.0001 0.0001 0.0001
amh 0.0027 0.0031 0.0033
ara 0.6469 0.6082 0.5639
asm 0.0016 0.0016 0.0020
aym 0.0000 0.0000 0.0000
aze 0.0451 0.0405 0.0387
bak 0.0010 0.0020 0.0012
bel 0.0158 0.0153 0.0134
ben 0.0857 0.0806 0.0799
bih 0.0004 0.0006 0.0006
bis 0.0001 0.0001 0.0001
bod 0.0010 0.0010 0.0009
bos 0.0439 0.0415 0.0387
bre 0.0026 0.0024 0.0024
bul 0.2556 0.2631 0.2546
cat 0.2332 0.2341 0.2234
ceb 0.0023 0.0020 0.0020
ces 0.9165 1.0039 0.9846
chr 0.0000 0.0000 0.0000
cos 0.0024 0.0024 0.0024
crs 0.0001 0.0001 0.0001
cym 0.0135 0.0120 0.0107
dan 0.4006 0.4180 0.4076
deu 5.4084 5.6032 5.4697
div 0.0020 0.0021 0.0019
dzo 0.0001 0.0001 0.0001
ell 0.5465 0.5396 0.5148
eng 46.2521 46.4560 47.8124
epo 0.0100 0.0086 0.0080
est 0.1354 0.1353 0.1200
eus 0.0326 0.0313 0.0299
fao 0.0033 0.0037 0.0037
fas 0.5881 0.6130 0.5835
fij 0.0001 0.0002 0.0001
fin 0.3825 0.3947 0.3848
fra 4.3941 4.4552 4.2907
fry 0.0033 0.0034 0.0027
gla 0.0028 0.0018 0.0016
gle 0.0058 0.0057 0.0059
glg 0.0408 0.0402 0.0369
glv 0.0003 0.0004 0.0004
got 0.0000 0.0000 0.0000
grn 0.0005 0.0007 0.0006
guj 0.0114 0.0111 0.0112
hat 0.0016 0.0013 0.0014
hau 0.0032 0.0027 0.0024
haw 0.0006 0.0007 0.0006
heb 0.2399 0.2347 0.2307
hin 0.1325 0.1279 0.1324
hmn 0.0007 0.0006 0.0005
hrv 0.1970 0.1950 0.1871
hun 0.5428 0.5480 0.5465
hye 0.0328 0.0300 0.0284
ibo 0.0006 0.0006 0.0005
iku 0.0002 0.0002 0.0002
ile 0.0003 0.0004 0.0003
ina 0.0006 0.0006 0.0005
ind 0.7630 0.7605 0.7036
ipk 0.0001 0.0001 0.0001
isl 0.0365 0.0378 0.0369
ita 2.3929 2.3877 2.3328
jav 0.0019 0.0019 0.0019
jpn 4.2860 4.7738 4.8690
kal 0.0011 0.0010 0.0008
kan 0.0108 0.0103 0.0102
kas 0.0000 0.0000 0.0000
kat 0.0393 0.0362 0.0320
kaz 0.0220 0.0212 0.0198
kha 0.0002 0.0002 0.0002
khm 0.0085 0.0083 0.0089
kin 0.0020 0.0017 0.0017
kir 0.0054 0.0049 0.0045
kor 0.7362 0.6723 0.5835
kur 0.0057 0.0056 0.0054
lao 0.0037 0.0037 0.0034
lat 0.0726 0.0996 0.1078
lav 0.0911 0.0825 0.0774
lif 0.0000 0.0000 0.0000
lin 0.0002 0.0002 0.0002
lit 0.1750 0.1568 0.1564
ltz 0.0033 0.0033 0.0034
lug 0.0001 0.0001 0.0001
mal 0.0193 0.0192 0.0201
mar 0.0192 0.0185 0.0188
mfe 0.0001 0.0001 0.0001
mkd 0.0321 0.0305 0.0291
mlg 0.0023 0.0023 0.0020
mlt 0.0054 0.0042 0.0035
mon 0.0172 0.0160 0.0166
mri 0.0015 0.0012 0.0013
msa 0.0731 0.0768 0.0719
mya 0.0109 0.0109 0.0096
nau 0.0000 0.0000 0.0000
nep 0.0271 0.0273 0.0262
nld 1.6558 1.7735 1.8066
nno 0.0160 0.0161 0.0160
nor 0.3235 0.3188 0.3193
nso 0.0001 0.0001 0.0001
nya 0.0007 0.0005 0.0005
oci 0.0027 0.0032 0.0029
ori 0.0048 0.0035 0.0035
orm 0.0004 0.0004 0.0004
pan 0.0050 0.0051 0.0050
pol 1.5780 1.6081 1.5477
por 2.1088 2.0693 2.1311
pus 0.0035 0.0028 0.0024
que 0.0005 0.0006 0.0006
roh 0.0012 0.0009 0.0009
ron 0.5988 0.5835 0.5278
run 0.0001 0.0001 0.0001
rus 5.9948 6.2413 6.0612
sag 0.0000 0.0000 0.0001
san 0.0025 0.0021 0.0027
sco 0.0005 0.0005 0.0005
sin 0.0083 0.0077 0.0067
slk 0.3320 0.3399 0.3352
slv 0.1486 0.1586 0.1533
smo 0.0006 0.0005 0.0004
sna 0.0006 0.0005 0.0006
snd 0.0012 0.0010 0.0011
som 0.0042 0.0042 0.0040
sot 0.0006 0.0004 0.0005
spa 4.1590 4.2539 4.2394
sqi 0.0403 0.0368 0.0359
srp 0.2052 0.2049 0.1910
ssw 0.0000 0.0000 0.0000
sun 0.0011 0.0010 0.0010
sux 0.0000 0.0000 0.0000
swa 0.0081 0.0075 0.0069
swe 0.6645 0.6822 0.6698
syr 0.0008 0.0006 0.0005
tam 0.0423 0.0402 0.0407
tat 0.0073 0.0080 0.0072
tel 0.0169 0.0165 0.0178
tgk 0.0040 0.0042 0.0045
tgl 0.0087 0.0083 0.0067
tha 0.3884 0.3781 0.3584
tir 0.0005 0.0007 0.0009
ton 0.0001 0.0001 0.0000
tsn 0.0001 0.0001 0.0001
tso 0.0001 0.0001 0.0000
tuk 0.0026 0.0021 0.0021
tur 0.8845 0.8550 0.9024
uig 0.0011 0.0010 0.0012
ukr 0.3865 0.4023 0.3793
urd 0.0272 0.0257 0.0261
uzb 0.0156 0.0148 0.0143
ven 0.0000 0.0000 0.0000
vie 0.7869 0.8379 0.8815
vol 0.0009 0.0009 0.0008
war 0.0029 0.0027 0.0025
wol 0.0001 0.0002 0.0001
xho 0.0029 0.0022 0.0014
yid 0.0018 0.0016 0.0014
yor 0.0011 0.0011 0.0011
zha 0.0000 0.0001 0.0001
zho 5.2986 4.8282 4.4734
zul 0.0029 0.0024 0.0012