PyPI package rot
Investigating the state of package rot on PyPI.
Introduction
The objective of this dataset is to provide a screenshot of the state of PyPI packages, so to facilitate investigations on the state of the Python ecosystem. While its first intended goal is to facilitate the identification of package rot, it can be used for other purposes as well.
At this time, we are building the dataset still and we are about 200k packages of 600k. We expect to have the dataset ready by the end of 2024.
Installing
As usual, you can install the package with pip:
pip install pypi-package-rot
CLI
The package provides the following CLI utilities to allow you to replicate the dataset. Do keep in mind that this operation will take a long time, as it involves scraping informations from websites that allow about 1 request per second.
Perpetual scraper
This command will scrape PyPI packages and store their metadata in the cache directory. Since the API RATE allows only about 1 request per second, it will take a long time (currently about 8 days) to retrieve all the packages. This utility will run in perpetuity, running and retrieving new packages continuously.
The email is needed to build an adequate user-agent string for the requests, so that PyPI can contact you if you are abusing the API.
pypi_package_rot perpetual_scraper --email "your@email.com"
Build the dataset
After having downloaded for a while the metadata of the packages, you can build the summarized anonymized dataset with the following command:
pypi_package_rot perpetual_builder --verbose --output "pypi.rot.v1.summarized.csv" --email "your@email.com"
Alternatively, to include all informations, you can use the following command:
pypi_package_rot perpetual_builder --verbose --full --output "pypi.rot.v1.full.json" --email "your@email.com"
Part of the procedure included testing whether the URLs in the metadata are still valid. This is done by sending a HEAD request to the URL and checking the status code. Such operations may take a long time (about one month at the time of writing). As per the previous command, the email is needed to build an adequate user-agent string for the requests, so that websites can contact you if you are abusing their services.
Contributing
Do you have some ideas to improve this dataset or associated analysis? Feel free to open an issue or a pull request.
License
This project is licensed under the MIT License - see the LICENSE file for details.
Metadata
Release files for pypi-package-rot 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| pypi_package_rot-0.1.0.tar.gz | 12.8 kB | Without provenance | Details for pypi_package_rot-0.1.0.tar.gz |
Release files / pypi_package_rot-0.1.0.tar.gz
| Download URL | pypi_package_rot-0.1.0.tar.gz |
|---|---|
| Size | 12.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
d9c68064eeb6c838fa2a252c61fc6953fd1161559c45f8ada03e712c884a860b
|
|
BLAKE2b-256 checksum How to use checksums |
17d64b228f530377f57f6448c730572bc5bd319312e60698159c5e53588e56df
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.0.0 CPython/3.11.5
|
Security for pypi-package-rot 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.