Web Graph Statistics

Description

This webpage presents operational statistics derived from Common Crawl's Web Graph releases, which show the structure and connectivity of the web as captured in the crawl releases. The data consists of host- and domain-level graphs, where hostnames are formatted in reverse domain name notation. These graphs include all types of links, such as those pointing to images, JS libraries, web fonts, and so on. However, only hostnames with valid IANA top-level domains are considered, excluding URLs that use IP addresses as host components.

The domain-level graphs are constructed by aggregating host-level data at the pay-level domain (PLD) level, using the public suffix list maintained on publicsuffix.org. This methodology provides a comprehensive view of the web's hierarchical structure, which is useful for research in areas like ranking algorithms, graph analysis, and link spam detection.

For those interested in exploring the Web Graphs, we provide tools and instructions through the cc-webgraph project on GitHub. We also have a Jupyter Notebook in our cc-notebooks repository. Some related papers on how these Web Graphs can be used can be found in the Related Reading section below. Additionally, the list of graph releases is accessible via the graphinfo.json file.

The top ten highest ranked hosts and domains from each release are listed below. Ranks are derived from Harmonic Centrality, and we also show PageRank for comparison.