opendatasets
opendatasets is a Python library for downloading datasets from online sources like Kaggle and Google Drive using a simple Python command.
Installation
Install the library using pip:
pip install opendatasets --upgrade
Usage - Downloading a dataset
Datasets can be downloaded within a Jupyter notebook or Python script using the opendatasets.download helper function. Here's some sample code for downloading the US Elections Dataset:
import opendatasets as od
dataset_url = 'https://www.kaggle.com/tunguz/us-elections-dataset'
od.download('https://www.kaggle.com/tunguz/us-elections-dataset')
dataset_url can also point to a public Google Drive link or a raw file URL.
Kaggle Credentials
opendatasets uses the Kaggle Official API for donwloading dataset from Kaggle. Follow these steps to find your API credentials:
-
Sign in to https://kaggle.com/, then click on your profile picture on the top right and select "My Account" from the menu.
-
Scroll down to the "API" section and click "Create New API Token". This will download a file
kaggle.jsonwith the following contents:
{"username":"YOUR_KAGGLE_USERNAME","key":"YOUR_KAGGLE_KEY"}
- When you run
opendatsets.download, you will be asked to enter your username & Kaggle API, which you can get from the file downloaded in step 2.
Note that you need to download the kaggle.json file only once. You can also place the kaggle.json file in the same directory as the Jupyter notebook, and the credentials will be read automatically.
Some interesting datasets
You can find interesting datasets on Kaggle: https://www.kaggle.com/datasets
You can also create a new dataset on Kaggle by uploading a CSV file here: https://www.kaggle.com/datasets?new=true (make sure to keep your dataset public, otherwise it will not be downloadable)
- Video Games sales: https://www.kaggle.com/gregorut/videogamesales
- World University Rankings: https://www.kaggle.com/mylesoneill/world-university-rankings
- Netflix Tv shows and Movies: https://www.kaggle.com/shivamb/netflix-shows/notebooks
- StackOverflow Developer Survey: https://www.kaggle.com/stackoverflow/stack-overflow-2018-developer-survey
- Google Play Store Android Apps Data: https://www.kaggle.com/lava18/google-play-store-apps
- Indian Stock Market Data: https://www.kaggle.com/rohanrao/nifty50-stock-market-data
- Indian Air Quality: https://www.kaggle.com/rohanrao/air-quality-data-in-india
- Worldwide Covid-19 Cases: https://www.kaggle.com/imdevskp/corona-virus-report
- USA Covid-19 Cases: https://www.kaggle.com/sudalairajkumar/covid19-in-usa
- US Election Results (2012): https://www.kaggle.com/tunguz/us-elections-dataset
- US Stock Market: https://www.kaggle.com/borismarjanovic/price-volume-data-for-all-us-stocks-etfs/
- Crop production in India: https://www.kaggle.com/srinivas1/agricuture-crops-production-in-india
- Agricultural raw material prices: https://www.kaggle.com/kianwee/agricultural-raw-material-prices-19902020
- Agricultural land values: https://www.kaggle.com/jmullan/agricultural-land-values-19972017
- Digital payments in India: https://www.kaggle.com/lazycipher/upi-usage-statistics-aug16-to-feb20
- US Unemployment Rate Data: https://www.kaggle.com/jayrav13/unemployment-by-county-us
- India Road accident Data: https://community.data.gov.in/statistics-of-road-accidents-in-india/
- Data Science Jobs Data:
- Youtube Trending Videos: https://www.kaggle.com/datasnaek/youtube-new
- Asteroid Dataset: https://www.kaggle.com/sakhawat18/asteroid-dataset
- Solar flares Data: https://www.kaggle.com/khsamaha/solar-flares-rhessi
- F-1 Race Data: https://www.kaggle.com/cjgdev/formula-1-race-data-19502017
- Automobile Insurance: https://www.kaggle.com/aashishjhamtani/automobile-insurance
- PUBG video game matches: https://www.kaggle.com/skihikingkevin/pubg-match-deaths
- CounterStrike GO (video game)
- Dota 2 (video game): https://www.kaggle.com/devinanzelmo/dota-2-matches
- Cricket One-Day Internationals Data: https://www.kaggle.com/jaykay12/odi-cricket-matches-19712017
- Cricket Indian Premier League Data: https://www.kaggle.com/nowke9/ipldata
- Basketball (NCAA): https://www.kaggle.com/ncaa/ncaa-basketball
- Basketball NBA Players Stats: https://www.kaggle.com/ncaa/ncaa-basketball
- Football datasets:
- Hotel Booking Demand: https://www.kaggle.com/jessemostipak/hotel-booking-demand
- New York Airbnb listings: https://www.kaggle.com/dgomonov/new-york-city-airbnb-open-data
Other sources to look for datasets:
If you use an external source other than Kaggle, you'll create a new dataset on Kaggle by uploading a CSV file here: https://www.kaggle.com/datasets?new=true (make sure to keep your dataset public, otherwise it will not be downloadable using opendatasets)
Curated Datasets
opendatasets also provides some curated datsets that you can download by passing the Dataset ID to opendatasets.download. Here's an example:
import opendatasets
opendatasets.download('stackoverflow-developer-survey-2020')
The following datasets are available for download.
| Dataset ID | Description | Source |
|---|---|---|
stackoverflow-developer-survey-2020 |
Stack Overflow Developer Survey 2020 | Stack Overflow |
owid-covid-19-latest |
Covid-19 Stats by Our World in Data | Our World in Data |
state-of-javascript-2016 |
State of Javascript Annual Survey 2016 | StateOfJS |
state-of-javascript-2017 |
State of Javascript Annual Survey 2017 | StateOfJS |
state-of-javascript-2018 |
State of Javascript Annual Survey 2018 | StateOfJS |
state-of-javascript-2019 |
State of Javascript Annual Survey 2019 | StateOfJS |
countries-languages-spoken |
Languages Spoken in Different Countries | Infoplease |
More datasets will be added soon..
Contributing
This is an open source project and we welcome contributions.
Local Development Setup
- Clone the repository:
git clone https://github.com/JovianML/opendatasets.git
- Setup the Python environment for development
conda create -n opendatasets python=3.5
conda activate opendatasets
pip install -r requirements.txt
- Open up the project in VS code and make your changes. Make sure to install the Python Extension for VS Code and select the
opendatasetsconda environment.
This package is developed and maintained by the Jovian team.
1 maintainer
Metadata
1 maintainer
Release files for opendatasets 0.1.22
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| opendatasets-0.1.22.tar.gz | 10.0 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| opendatasets-0.1.22-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 25.4 kB
Release files / opendatasets-0.1.22.tar.gz
| Download URL | opendatasets-0.1.22.tar.gz |
|---|---|
| Size | 10.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
52b2e0c1cc80d9f44842e3373532d92683f7c0f5c3e72b3f1f3e2750d30da4db
|
|
BLAKE2b-256 checksum How to use checksums |
1a09d833ab8037b6482166373fadd166067615b2f1c55df3f97c1f3657ee19ca
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.15.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/40.2.0 requests-toolbelt/0.9.1 tqdm/4.49.0 CPython/3.5.6
|
Release files / opendatasets-0.1.22-py3-none-any.whl
| Download URL | opendatasets-0.1.22-py3-none-any.whl |
|---|---|
| Size | 15.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
8d85a6d32fd7831672eddcae366a2488b5b5b5837433c4db02152e38b50e70c9
|
|
BLAKE2b-256 checksum How to use checksums |
00e712300c2f886b846375c78a4f32c0ae1cd20bdcf305b5ac45b8d7eceda3ec
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.15.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/40.2.0 requests-toolbelt/0.9.1 tqdm/4.49.0 CPython/3.5.6
|
Security for opendatasets 0.1.22
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
