Load data about Python packages from PyPI into SQLite
Project description
pypi-to-sqlite
Load data about Python packages from PyPI into SQLite
Installation
Install this tool using pip
:
pip install pypi-to-sqlite
Usage
To create a SQLite database with details of one or more packages, run:
pypi-to-sqlite pypi.db datasette sqlite-utils
You can also process JSON that you have previously saved to disk like so:
curl -o datasette.json https://pypi.org/pypi/datasette/json
pypi-to-sqlite pypi.db -f datasette.json
Database schema
CREATE TABLE [packages] (
[name] TEXT PRIMARY KEY,
[summary] TEXT,
[classifiers] TEXT,
[description] TEXT,
[author] TEXT,
[author_email] TEXT,
[description_content_type] TEXT,
[home_page] TEXT,
[keywords] TEXT,
[license] TEXT,
[maintainer] TEXT,
[maintainer_email] TEXT,
[package_url] TEXT,
[platform] TEXT,
[project_url] TEXT,
[project_urls] TEXT,
[release_url] TEXT,
[requires_dist] TEXT,
[requires_python] TEXT,
[version] TEXT,
[yanked] INTEGER,
[yanked_reason] TEXT
);
CREATE TABLE [versions] (
[id] TEXT PRIMARY KEY,
[package] TEXT REFERENCES [packages]([name]),
[name] TEXT
);
CREATE TABLE [releases] (
[package] TEXT REFERENCES [packages]([name]),
[version] TEXT REFERENCES [versions]([id]),
[packagetype] TEXT,
[filename] TEXT,
[comment_text] TEXT,
[digests] TEXT,
[has_sig] INTEGER,
[md5_digest] TEXT PRIMARY KEY,
[python_version] TEXT,
[requires_python] TEXT,
[size] INTEGER,
[upload_time] TEXT,
[upload_time_iso_8601] TEXT,
[url] TEXT,
[yanked] INTEGER,
[yanked_reason] TEXT
);
pypi-to-sqlite --help
Usage: pypi-to-sqlite [OPTIONS] DB_PATH [PACKAGE]...
Load data about Python packages from PyPI into SQLite
Usage example:
pypi-to-sqlite pypy.db datasette sqlite-utils
Use -f to load data from a JSON file instead:
pypi-to-sqlite pypy.db -f datasette.json
Options:
--version Show the version and exit.
-f, --file FILENAME Import JSON from this file
-d, --delay FLOAT Wait this many seconds between requests
--help Show this message and exit.
Development
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd pypi-to-sqlite
python -m venv venv
source venv/bin/activate
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
pypi-to-sqlite-0.1.tar.gz
(7.7 kB
view hashes)
Built Distribution
Close
Hashes for pypi_to_sqlite-0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 15aeb5be7cc89ceafeb2d163e1f97569046c043afc4c352040ab464b94376277 |
|
MD5 | 1e3948cdad96b653fc0e8efe497d9b99 |
|
BLAKE2b-256 | f303ea934d6eb5064002abbde3a8b3cf228de56a0bec20ff5b2e2a081777e1bb |