crates-miner 1.0.2

Mine and extract complete package lists from crates.io registry

Crates.io Package Miner

A Python tool to mine and extract complete package lists from the crates.io (Rust) registry.

Features

• Downloads official crates.io database dump (~1000 MB)
• Extracts package metadata including homepage and repository URLs
• Progress tracking with visual feedback
• Automatic cleanup of temporary files
• Outputs standardized CSV format for cross-ecosystem analysis

Installation

pip install crates-miner

Quick Start

crates-miner

Or use as a Python module:

from crates_miner import mine_crates
mine_crates()

Output

Generates a CSV file with crate information:

• Package ID, Platform, Name
• Homepage URL, Repository URL

Performance

• Runtime: Fast (depends on download speed)
• No API rate limits (uses database dump)
• Processes ~100,000+ crates
• Download size: ~1000 MB compressed

Data Source

• Crates.io Database Dump: https://static.crates.io/db-dump.tar.gz

License

MIT License - see LICENSE file for details

Processing Details

Database Dump Structure

The downloaded archive contains a dated directory (e.g., 2025-11-03-020107) with:

2025-11-03-020107/
├── data/
│   ├── crates.csv           ← Used by this script
│   ├── versions.csv
│   ├── dependencies.csv
│   ├── teams.csv
│   └── ... (other files)
└── metadata/
    └── ... (metadata files)

Extraction Process

1. Download: Fetches db-dump.tar.gz from static.crates.io
2. Extract: Decompresses to Code/Script/Crates-Miner/crates-db/
3. Process: Reads data/crates.csv from the dated directory
4. Transform: Converts to standardized format
5. Output: Writes to Resource/Package/Package-List/Crates_New.csv
6. Cleanup: Deletes the .tar.gz file (keeps extracted data)

Data Transformation

The script transforms crates.io data to match the cross-ecosystem format:

Input (crates.csv):

id,name,homepage,repository,description,downloads,...
12345,serde,https://serde.rs,https://github.com/serde-rs/serde,"A serialization framework",50000000,...

Output (Crates_New.csv):

ID,Platform,Name,Homepage URL,Repository URL
1,Crates.io,serde,https://serde.rs,https://github.com/serde-rs/serde

Files

• mine_crates.py: Main script
• requirements.txt: Python dependencies (requests, pandas, tqdm)
• setup.sh: Automated setup script
• crates-db/: Temporary directory for extracted database (created during execution)
• Output: ../../../Resource/Package/Package-List/Crates_New.csv

Troubleshooting

"Could not download database dump"

Check that:

• You have internet connectivity
• crates.io is accessible: curl -I https://static.crates.io/db-dump.tar.gz
• You have sufficient disk space (~500 MB for extraction)

"crates.csv not found"

This may occur if:

• The database dump structure has changed
• Extraction failed partway through
• The archive is corrupted

Solution: Delete crates-db/ directory and run again to re-download.

"Permission denied" when creating output directory

Ensure you have write permissions to:

• Current directory (for temporary files)
• Resource/Package/Package-List/ (for output)

"Memory error" during processing

The crates.io database is large (100K+ crates). If you encounter memory issues:

• Close other applications
• Increase available system memory
• Consider processing in chunks (requires script modification)

Virtual environment issues

If you encounter errors related to the virtual environment:

1. Delete the venv folder: rm -rf venv
2. Re-run the setup script: ./setup.sh
3. Virtual environments cannot be moved after creation - recreate if you move the directory

---

Code Explanation

Architecture Overview

The Crates.io Miner is a simpler tool compared to the Directory Structure Miners, focused on a single task: downloading and processing the official crates.io database dump.

Key characteristics:

• Batch Processing: Downloads entire database at once
• Official Source: Uses crates.io's public database dump
• No API Calls: Works with static data dump (no rate limiting)
• Large Scale: Processes 100K+ packages

1. Download Function

def download_file(url, filename):
    """Downloads a file from a URL with a progress bar."""
    response = requests.get(url, stream=True)
    response.raise_for_status()
    total_size = int(response.headers.get("content-length", 0))
    block_size = 1024  # 1 Kilobyte

    with open(filename, "wb") as f, tqdm(
        desc=filename,
        total=total_size,
        unit="iB",
        unit_scale=True,
        unit_divisor=1024,
    ) as bar:
        for data in response.iter_content(block_size):
            bar.update(len(data))
            f.write(data)

Purpose: Downloads large files with progress tracking.

Features:

• Streaming: Uses stream=True to avoid loading entire file in memory
• Progress Bar: Shows download progress with tqdm
• Chunk Processing: Downloads in 1KB chunks
• Size Display: Shows human-readable units (MB, GB)

Process:

1. Make HTTP GET request with streaming
2. Get total file size from headers
3. Open file in binary write mode
4. Download and write in chunks
5. Update progress bar after each chunk

2. Main Mining Function

def mine_crates():
    """Mines crates.io to get the whole list of Rust packages from the database dump."""

    dump_url = "https://static.crates.io/db-dump.tar.gz"
    dump_path = "db-dump.tar.gz"
    extract_path = "Code/Script/Crates-Miner/crates-db"

Purpose: Orchestrates the entire mining process.

Configuration:

• dump_url: Official crates.io database dump URL
• dump_path: Local filename for downloaded archive
• extract_path: Directory for extracted files

3. Download Phase

    # Download the database dump
    if not os.path.exists(dump_path):
        print("Downloading crates.io database dump...")
        download_file(dump_url, dump_path)
    else:
        print("Database dump already downloaded.")

Logic:

• Checks if archive already exists
• Skips download if file present (saves time on reruns)
• Downloads ~100-200 MB compressed file

Why This Matters:

• Avoids re-downloading large file unnecessarily
• Useful during development/testing
• Saves bandwidth and time

4. Extraction Phase

    # Extract the database dump
    print("Extracting database dump...")
    with tarfile.open(dump_path, "r:gz") as tar:
        tar.extractall(path=extract_path)

    # Delete the tar.gz file
    if os.path.exists(dump_path):
        print("Deleting database dump archive...")
        os.remove(dump_path)

Process:

1. Open Archive: Opens .tar.gz file in read mode
2. Extract All: Extracts to crates-db/ directory
3. Cleanup: Removes archive to save disk space

Tar Format: r:gz means read mode with gzip compression

5. Directory Discovery

    # Find the actual data directory (it has a date in the name)
    data_dir = ""
    for item in os.listdir(extract_path):
        if os.path.isdir(os.path.join(extract_path, item)):
            data_dir = os.path.join(extract_path, item)
            break

    if not data_dir:
        print("Could not find data directory in the extracted archive.")
        return

Purpose: Finds the dated directory containing the actual data.

Why Dynamic: The database dump directory name changes with each snapshot:

• 2025-11-03-020107/
• 2025-11-04-020107/
• etc.

Logic:

1. List all items in extraction path
2. Find first directory (not file)
3. Use that as data directory
4. Error if no directory found

6. CSV Path Construction

    crates_csv_path = os.path.join(data_dir, "data", "crates.csv")
    if not os.path.exists(crates_csv_path):
        print(f"crates.csv not found in {data_dir}")
        return

Path Structure: crates-db/{date}/data/crates.csv

Validation: Checks file exists before attempting to read

7. Data Processing

    print("Processing crate data...")
    # Read the crates data
    df = pd.read_csv(crates_csv_path)

    # Create the path to the output file
    output_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..', '..', 'Resource', 'Package', 'Package-List'))
    if not os.path.exists(output_dir):
        os.makedirs(output_dir)
    output_file = os.path.join(output_dir, "Crates_New.csv")

Reading:

• Uses pandas read_csv() for efficient processing
• Automatically handles CSV parsing and data types

Output Path:

• Navigates up 3 directories from script location
• Ensures output directory exists (makedirs)
• Constructs full path to output file

Path Navigation:

Current: Code/Script/Crates-Miner/mine_crates.py
Up 1:    Code/Script/Crates-Miner/
Up 2:    Code/Script/
Up 3:    Code/
Result:  Code/../../../Resource/Package/Package-List/

8. CSV Writing

    with open(output_file, "w", newline="", encoding="utf-8") as f:
        writer = csv.writer(f)
        writer.writerow(["ID", "Platform", "Name", "Homepage URL", "Repository URL"])

        for index, row in tqdm(df.iterrows(), total=df.shape[0], desc="Writing to CSV"):
            writer.writerow([
                index + 1,
                "Crates.io",
                row["name"],
                row["homepage"],
                row["repository"],
            ])

    print(f"Successfully saved {df.shape[0]} crates to {output_file}")

Process:

1. Open File: In write mode with UTF-8 encoding
2. Write Header: Column names for standardized format
3. Iterate Rows: Loop through pandas DataFrame with progress bar
4. Transform Data: Convert each row to standard format
5. Write Row: Add to output CSV

Data Transformation:

• ID: Uses index + 1 (1-based instead of 0-based)
• Platform: Hardcoded as "Crates.io"
• Name: Direct mapping from row["name"]
• Homepage URL: Direct mapping from row["homepage"]
• Repository URL: Direct mapping from row["repository"]

Progress Tracking:

• tqdm() with total=df.shape[0] shows percentage complete
• Useful for large datasets (100K+ rows)
• Provides time estimates

Workflow Summary

┌─────────────────────────────────────────────────────────────┐
│                    Crates.io Miner Workflow                 │
└─────────────────────────────────────────────────────────────┘

1. Check if db-dump.tar.gz exists
   ├─ Yes → Skip download
   └─ No  → Download from static.crates.io (~200 MB)

2. Extract db-dump.tar.gz
   ├─ Decompress with gzip
   ├─ Extract tar to crates-db/
   └─ Delete archive file

3. Find dated directory
   ├─ Scan crates-db/
   └─ Locate {date}/data/crates.csv

4. Load data with pandas
   └─ Read entire crates.csv into DataFrame

5. Create output directory structure
   └─ Resource/Package/Package-List/

6. Transform and write CSV
   ├─ Header: ID, Platform, Name, Homepage URL, Repository URL
   ├─ For each crate:
   │  ├─ Generate sequential ID
   │  ├─ Set Platform = "Crates.io"
   │  ├─ Copy name, homepage, repository
   │  └─ Write row
   └─ Show progress bar

7. Complete
   └─ Print success message with count

Error Handling

The script includes basic error handling for common scenarios:

Download Errors:

response.raise_for_status()

• Raises exception for HTTP errors (404, 500, etc.)
• Stops execution if download fails

File Not Found:

if not os.path.exists(crates_csv_path):
    print(f"crates.csv not found in {data_dir}")
    return

• Checks for expected files
• Gracefully exits with error message

Directory Creation:

if not os.path.exists(output_dir):
    os.makedirs(output_dir)

• Creates output directory if missing
• Prevents write errors

---