Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives
The language of a document is identified by Compact Language Detector 2 (CLD2). It is able to identify 160 different languages and up to 3 languages per document. The table lists the percentage covered by the primary language of a document (returned first by CLD2). So far, only HTML pages are passed to the language detector.
| crawl | CC-MAIN-2018-34 |
|---|---|
| language | |
| <other> | 0.0000 |
| aar | 0.0002 |
| abk | 0.0002 |
| afr | 0.0077 |
| aka | 0.0001 |
| amh | 0.0022 |
| ara | 0.7106 |
| asm | 0.0005 |
| aym | 0.0001 |
| aze | 0.0361 |
| bak | 0.0012 |
| bel | 0.0126 |
| ben | 0.0482 |
| bih | 0.0006 |
| bis | 0.0001 |
| blu | 0.0007 |
| bod | 0.0011 |
| bos | 0.0298 |
| bre | 0.0014 |
| bul | 0.2770 |
| cat | 0.1811 |
| ceb | 0.0012 |
| ces | 1.0815 |
| chr | 0.0000 |
| cos | 0.0014 |
| crs | 0.0001 |
| cym | 0.0065 |
| dan | 0.3870 |
| deu | 5.2847 |
| div | 0.0007 |
| dzo | 0.0001 |
| ell | 0.5089 |
| eng | 46.6226 |
| epo | 0.0043 |
| est | 0.1040 |
| eus | 0.0192 |
| fao | 0.0028 |
| fas | 0.6730 |
| fij | 0.0001 |
| fin | 0.3755 |
| fra | 4.5931 |
| fry | 0.0022 |
| gla | 0.0015 |
| gle | 0.0039 |
| glg | 0.0233 |
| glv | 0.0003 |
| got | 0.0000 |
| grn | 0.0009 |
| guj | 0.0052 |
| hat | 0.0013 |
| hau | 0.0016 |
| haw | 0.0003 |
| heb | 0.1706 |
| hin | 0.0729 |
| hrv | 0.1421 |
| hun | 0.5155 |
| hye | 0.0239 |
| ibo | 0.0002 |
| iku | 0.0001 |
| ile | 0.0003 |
| ina | 0.0008 |
| ind | 0.7861 |
| ipk | 0.0001 |
| isl | 0.0324 |
| ita | 2.0861 |
| jav | 0.0012 |
| jpn | 5.6766 |
| kal | 0.0006 |
| kan | 0.0063 |
| kat | 0.0267 |
| kaz | 0.0143 |
| kha | 0.0002 |
| khm | 0.0052 |
| kin | 0.0018 |
| kir | 0.0034 |
| kor | 0.5940 |
| kur | 0.0043 |
| lao | 0.0020 |
| lat | 0.0352 |
| lav | 0.0769 |
| lin | 0.0002 |
| lit | 0.1366 |
| ltz | 0.0021 |
| lug | 0.0001 |
| mal | 0.0136 |
| mar | 0.0075 |
| mfe | 0.0001 |
| mkd | 0.0175 |
| mlg | 0.0021 |
| mlt | 0.0020 |
| mon | 0.0128 |
| mri | 0.0006 |
| msa | 0.1048 |
| mya | 0.0084 |
| nau | 0.0000 |
| nep | 0.0115 |
| nld | 1.4527 |
| nno | 0.0131 |
| nor | 0.3097 |
| nso | 0.0001 |
| nya | 0.0005 |
| oci | 0.0026 |
| ori | 0.0018 |
| orm | 0.0003 |
| pan | 0.0031 |
| pol | 1.7673 |
| por | 2.1048 |
| pus | 0.0023 |
| que | 0.0003 |
| roh | 0.0013 |
| ron | 0.5437 |
| run | 0.0001 |
| rus | 9.5047 |
| sag | 0.0000 |
| san | 0.0018 |
| sco | 0.0008 |
| sin | 0.0056 |
| slk | 0.2407 |
| slv | 0.1018 |
| smo | 0.0003 |
| sna | 0.0003 |
| snd | 0.0005 |
| som | 0.0045 |
| sot | 0.0002 |
| spa | 4.3270 |
| sqi | 0.0301 |
| srp | 0.1546 |
| ssw | 0.0001 |
| sun | 0.0006 |
| swa | 0.0067 |
| swe | 0.7680 |
| syr | 0.0003 |
| tam | 0.0317 |
| tat | 0.0051 |
| tel | 0.0088 |
| tgk | 0.0025 |
| tgl | 0.0083 |
| tha | 0.3376 |
| tir | 0.0007 |
| ton | 0.0001 |
| tsn | 0.0001 |
| tso | 0.0001 |
| tuk | 0.0009 |
| tur | 0.9263 |
| uig | 0.0009 |
| ukr | 0.3868 |
| urd | 0.0211 |
| uzb | 0.0099 |
| ven | 0.0000 |
| vie | 0.7932 |
| vol | 0.0006 |
| war | 0.0022 |
| wol | 0.0001 |
| xho | 0.0004 |
| yid | 0.0013 |
| yor | 0.0003 |
| zha | 0.0000 |
| zho | 4.4900 |
| zul | 0.0004 |