Skip to main content

Load data about Python packages from PyPI into SQLite

Project description

pypi-to-sqlite

PyPI Changelog Tests License

Load data about Python packages from PyPI into SQLite

Installation

Install this tool using pip:

pip install pypi-to-sqlite

Usage

To create a SQLite database with details of one or more packages, run:

pypi-to-sqlite pypi.db datasette sqlite-utils

You can also process JSON that you have previously saved to disk like so:

curl -o datasette.json https://pypi.org/pypi/datasette/json
pypi-to-sqlite pypi.db -f datasette.json

The tool will create three tables: packages, versions and releases. The full table schema is shown below.

To create the tables with a prefix, use --prefix prefix. For example:

pypi-to-sqlite pypi.db datasette --prefix pypi_

This will create tables called pypi_packages, pypi_versions and pypi_releases.

Database schema

CREATE TABLE [packages] (
   [name] TEXT PRIMARY KEY,
   [summary] TEXT,
   [classifiers] TEXT,
   [description] TEXT,
   [author] TEXT,
   [author_email] TEXT,
   [description_content_type] TEXT,
   [home_page] TEXT,
   [keywords] TEXT,
   [license] TEXT,
   [maintainer] TEXT,
   [maintainer_email] TEXT,
   [package_url] TEXT,
   [platform] TEXT,
   [project_url] TEXT,
   [project_urls] TEXT,
   [release_url] TEXT,
   [requires_dist] TEXT,
   [requires_python] TEXT,
   [version] TEXT,
   [yanked] INTEGER,
   [yanked_reason] TEXT
);
CREATE TABLE [versions] (
   [id] TEXT PRIMARY KEY,
   [package] TEXT REFERENCES [packages]([name]),
   [name] TEXT
);
CREATE TABLE [releases] (
   [md5_digest] TEXT PRIMARY KEY,
   [package] TEXT REFERENCES [packages]([name]),
   [version] TEXT REFERENCES [versions]([id]),
   [packagetype] TEXT,
   [filename] TEXT,
   [comment_text] TEXT,
   [digests] TEXT,
   [has_sig] INTEGER,
   [python_version] TEXT,
   [requires_python] TEXT,
   [size] INTEGER,
   [upload_time] TEXT,
   [upload_time_iso_8601] TEXT,
   [url] TEXT,
   [yanked] INTEGER,
   [yanked_reason] TEXT
);

pypi-to-sqlite --help

Usage: pypi-to-sqlite [OPTIONS] DB_PATH [PACKAGE]...

  Load data about Python packages from PyPI into SQLite

  Usage example:

      pypi-to-sqlite pypy.db datasette sqlite-utils

  Use -f to load data from a JSON file instead:

      pypi-to-sqlite pypy.db -f datasette.json

  Created tables will be packages, versions and releases

  To create tables called pypi_packages, pypi_versions, pypi_releases use
  --prefix pypi_:

      pypi-to-sqlite pypy.db datasette sqlite-utils --prefix pypi_

Options:
  --version            Show the version and exit.
  -f, --file FILENAME  Import JSON from this file
  -d, --delay FLOAT    Wait this many seconds between requests
  --prefix TEXT        Prefix to use for the created database tables
  --help               Show this message and exit.

Development

To contribute to this tool, first checkout the code. Then create a new virtual environment:

cd pypi-to-sqlite
python -m venv venv
source venv/bin/activate

Now install the dependencies and test dependencies:

pip install -e '.[test]'

To run the tests:

pytest

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pypi-to-sqlite-0.2.1.tar.gz (7.9 kB view hashes)

Uploaded source

Built Distribution

pypi_to_sqlite-0.2.1-py3-none-any.whl (8.6 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page