Load data about Python packages from PyPI into SQLite
Project description
pypi-to-sqlite
Load data about Python packages from PyPI into SQLite
Installation
Install this tool using pip
:
pip install pypi-to-sqlite
Usage
To create a SQLite database with details of one or more packages, run:
pypi-to-sqlite pypi.db datasette sqlite-utils
You can also process JSON that you have previously saved to disk like so:
curl -o datasette.json https://pypi.org/pypi/datasette/json
pypi-to-sqlite pypi.db -f datasette.json
The tool will create three tables: packages
, versions
and releases
. The full table schema is shown below.
To create the tables with a prefix, use --prefix prefix
. For example:
pypi-to-sqlite pypi.db datasette --prefix pypi_
This will create tables called pypi_packages
, pypi_versions
and pypi_releases
.
Database schema
CREATE TABLE [packages] (
[name] TEXT PRIMARY KEY,
[summary] TEXT,
[classifiers] TEXT,
[description] TEXT,
[author] TEXT,
[author_email] TEXT,
[description_content_type] TEXT,
[home_page] TEXT,
[keywords] TEXT,
[license] TEXT,
[maintainer] TEXT,
[maintainer_email] TEXT,
[package_url] TEXT,
[platform] TEXT,
[project_url] TEXT,
[project_urls] TEXT,
[release_url] TEXT,
[requires_dist] TEXT,
[requires_python] TEXT,
[version] TEXT,
[yanked] INTEGER,
[yanked_reason] TEXT
);
CREATE TABLE [versions] (
[id] TEXT PRIMARY KEY,
[package] TEXT REFERENCES [packages]([name]),
[name] TEXT
);
CREATE TABLE [releases] (
[md5_digest] TEXT PRIMARY KEY,
[package] TEXT REFERENCES [packages]([name]),
[version] TEXT REFERENCES [versions]([id]),
[packagetype] TEXT,
[filename] TEXT,
[comment_text] TEXT,
[digests] TEXT,
[has_sig] INTEGER,
[python_version] TEXT,
[requires_python] TEXT,
[size] INTEGER,
[upload_time] TEXT,
[upload_time_iso_8601] TEXT,
[url] TEXT,
[yanked] INTEGER,
[yanked_reason] TEXT
);
pypi-to-sqlite --help
Usage: pypi-to-sqlite [OPTIONS] DB_PATH [PACKAGE]...
Load data about Python packages from PyPI into SQLite
Usage example:
pypi-to-sqlite pypy.db datasette sqlite-utils
Use -f to load data from a JSON file instead:
pypi-to-sqlite pypy.db -f datasette.json
Created tables will be packages, versions and releases
To create tables called pypi_packages, pypi_versions, pypi_releases use
--prefix pypi_:
pypi-to-sqlite pypy.db datasette sqlite-utils --prefix pypi_
Options:
--version Show the version and exit.
-f, --file FILENAME Import JSON from this file
-d, --delay FLOAT Wait this many seconds between requests
--prefix TEXT Prefix to use for the created database tables
--help Show this message and exit.
Development
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd pypi-to-sqlite
python -m venv venv
source venv/bin/activate
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for pypi_to_sqlite-0.2.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 620a5f7eaca783aed4b8bf93f0864bc58056231d30493241485c7a19264ba072 |
|
MD5 | 4186b881fa8b72f11d781beabd8bc6b2 |
|
BLAKE2b-256 | bac65085b6ddb6db51093163054fff5f65257e80af462f81624ce4c039800675 |