pyccwebgraph: Python Interface to CommonCrawl Webgraph
Discover related domains using link topology from CommonCrawl's webgraph.
Installation
Prerequisites:
- Python 3.8+
- Java 17+ (install instructions)
- ~30GB disk space for webgraph data
pip install pyccwebgraph
First use downloads graph data:
from pyccwebgraph import CCWebgraph, get_available_versions
# List available versions
versions = get_available_versions()
print(versions[:3]) # ['cc-main-2024-nov-dec-jan', 'cc-main-2024-feb-apr-may', ...]
webgraph = CCWebgraph.setup(
webgraph_dir="/data/my-webgraph",
version="cc-main-2024-feb-apr-may"
)
# Find domains that link TO seeds (backlinks)
results = webgraph.discover_backlinks(
seeds=["cnn.com", "bbc.com", "nytimes.com"],
min_connections=3 # Must link to all seeds
)
print(f"Found {len(results['nodes'])} domains")
print(f"Top result: {results['nodes'][0]}")
# {'domain': 'news-aggregator.com', 'connections': 15, 'percentage': 50.0}
Working with NetworkX
# Get results as NetworkX graph
G = webgraph.discover_backlinks(
seeds=["cnn.com", "bbc.com"],
min_connections=2,
format='networkx' # Returns nx.DiGraph
)
# Run standard NetworkX algorithms
import networkx as nx
# Centrality analysis
pr = nx.pagerank(G)
bc = nx.betweenness_centrality(G)
# Community detection
from cdlib import algorithms
communities = algorithms.louvain(G)
# Visualization
from pyvis.network import Network
net = Network(notebook=True)
net.from_nx(G)
net.show("network.html")
Performance: Large Graphs with NetworKit
For large discovered subgraphs (>100K nodes), use NetworKit instead of NetworkX:
# Discover large subgraph
G_nk, name_map = webgraph.discover_backlinks(
seeds=seed_list,
min_connections=2,
format='networkit' # Returns NetworKit graph
)
CC-Webgraph mapping
# Check if domain exists in graph
vid = webgraph.domain_to_id("example.com")
if vid is not None:
print(f"Found at vertex ID {vid}")
# Get all domains this domain links to
outlinks = webgraph.get_successors("cnn.com")
print(f"CNN links to {len(outlinks)} domains")
# Get all domains linking to this domain
backlinks = webgraph.get_predecessors("cnn.com")
print(f"{len(backlinks)} domains link to CNN")
# Validate seeds before discovery
found, missing = webgraph.validate_seeds(["cnn.com", "fake-site.xyz"])
print(f"Found: {found}")
print(f"Missing: {missing}")
Links
- Interactive demo: https://github.com/PeterCarragher/NetNeighbors
- PyPI: https://pypi.org/project/pyccwebgraph/
- Documentation: https://pyccwebgraph.readthedocs.io/
- Research Papers for webgraph-based discovery:
- CommonCrawl Webgraphs: https://commoncrawl.org/web-graphs
- cc-webgraph: https://github.com/commoncrawl/cc-webgraph
Metadata
Release files for pyccwebgraph 0.3.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| pyccwebgraph-0.3.3.tar.gz | 19.0 kB | Without provenance | Details for pyccwebgraph-0.3.3.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| pyccwebgraph-0.3.3-py3-none-any.whl | Python 3 | none | any | Details for pyccwebgraph-0.3.3-py3-none-any.whl |
Total release size: 36.1 kB
Release files / pyccwebgraph-0.3.3.tar.gz
| Download URL | pyccwebgraph-0.3.3.tar.gz |
|---|---|
| Size | 19.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
75fff40f750f4d06464023d586ff212d7c338744e11f519453c32bc1a646ca18
|
|
BLAKE2b-256 checksum How to use checksums |
29efefee2b2be6e5e2e2899a9fe5ed2030681025aff5d1af7882e70c0bc6e20a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Release files / pyccwebgraph-0.3.3-py3-none-any.whl
| Download URL | pyccwebgraph-0.3.3-py3-none-any.whl |
|---|---|
| Size | 17.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
0ebedd46fa916e7fdbd1eae3902b7aea11ff0919c0d4ac91a0d63b3c7de8a5c1
|
|
BLAKE2b-256 checksum How to use checksums |
38dcad7a25d002690e2fa7490faa81a306f931db0fe4c8d87a6bf896b751eb43
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Security for pyccwebgraph 0.3.3
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.