Skip to main content

pyccwebgraph: Python Interface to CommonCrawl Webgraph

PyPI version Python 3.8+ License: MIT

Discover related domains using link topology from CommonCrawl's webgraph.

Installation

Prerequisites:

pip install pyccwebgraph

First use downloads graph data:

from pyccwebgraph import CCWebgraph, get_available_versions

# List available versions
versions = get_available_versions()
print(versions[:3])  # ['cc-main-2024-nov-dec-jan', 'cc-main-2024-feb-apr-may', ...]

webgraph = CCWebgraph.setup(
    webgraph_dir="/data/my-webgraph",
    version="cc-main-2024-feb-apr-may"
)

# Find domains that link TO seeds (backlinks)
results = webgraph.discover_backlinks(
    seeds=["cnn.com", "bbc.com", "nytimes.com"],
    min_connections=3  # Must link to all seeds
)

print(f"Found {len(results['nodes'])} domains")
print(f"Top result: {results['nodes'][0]}")
# {'domain': 'news-aggregator.com', 'connections': 15, 'percentage': 50.0}

Working with NetworkX

# Get results as NetworkX graph
G = webgraph.discover_backlinks(
    seeds=["cnn.com", "bbc.com"],
    min_connections=2,
    format='networkx'  # Returns nx.DiGraph
)

# Run standard NetworkX algorithms
import networkx as nx

# Centrality analysis
pr = nx.pagerank(G)
bc = nx.betweenness_centrality(G)

# Community detection
from cdlib import algorithms
communities = algorithms.louvain(G)

# Visualization
from pyvis.network import Network
net = Network(notebook=True)
net.from_nx(G)
net.show("network.html")

Performance: Large Graphs with NetworKit

For large discovered subgraphs (>100K nodes), use NetworKit instead of NetworkX:

# Discover large subgraph
G_nk, name_map = webgraph.discover_backlinks(
    seeds=seed_list,
    min_connections=2,
    format='networkit'  # Returns NetworKit graph
)

CC-Webgraph mapping

# Check if domain exists in graph
vid = webgraph.domain_to_id("example.com")
if vid is not None:
    print(f"Found at vertex ID {vid}")

# Get all domains this domain links to
outlinks = webgraph.get_successors("cnn.com")
print(f"CNN links to {len(outlinks)} domains")

# Get all domains linking to this domain  
backlinks = webgraph.get_predecessors("cnn.com")
print(f"{len(backlinks)} domains link to CNN")

# Validate seeds before discovery
found, missing = webgraph.validate_seeds(["cnn.com", "fake-site.xyz"])
print(f"Found: {found}")
print(f"Missing: {missing}")

Metadata

Release files for pyccwebgraph 0.3.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pyccwebgraph 0.3.3
File Size Uploaded Provenance Actions
pyccwebgraph-0.3.3.tar.gz 19.0 kB Without provenance Details for pyccwebgraph-0.3.3.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for pyccwebgraph 0.3.3
File Interpreter ABI Platform Actions
pyccwebgraph-0.3.3-py3-none-any.whl Python 3 none any Details for pyccwebgraph-0.3.3-py3-none-any.whl

Total release size: 36.1 kB

Release files / pyccwebgraph-0.3.3.tar.gz

Download URL pyccwebgraph-0.3.3.tar.gz
Size 19.0 kB
Tags Source
SHA-256 checksum
How to use checksums
75fff40f750f4d06464023d586ff212d7c338744e11f519453c32bc1a646ca18
BLAKE2b-256 checksum
How to use checksums
29efefee2b2be6e5e2e2899a9fe5ed2030681025aff5d1af7882e70c0bc6e20a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.16

Release files / pyccwebgraph-0.3.3-py3-none-any.whl

Download URL pyccwebgraph-0.3.3-py3-none-any.whl
Size 17.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
0ebedd46fa916e7fdbd1eae3902b7aea11ff0919c0d4ac91a0d63b3c7de8a5c1
BLAKE2b-256 checksum
How to use checksums
38dcad7a25d002690e2fa7490faa81a306f931db0fe4c8d87a6bf896b751eb43
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.16

Security for pyccwebgraph 0.3.3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.3.3 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page