Dapla Toolbelt
Python module for use within Jupyterlab notebooks, specifically aimed for Statistics Norway's data platform called
Dapla. It contains support for authenticated access to Google Services such as Google Cloud Storage (GCS) and custom
Dapla services such as Maskinporten Guardian.
Features
These operations are supported:
- List contents of a bucket
- Open a file in GCS
- Copy a file from GCS into local
- Load a file (CSV, JSON or XML) from GCS into a pandas dataframe
- Save contents of a data frame into a file (CSV, JSON, XML) in GCS
When the user gives the path to a resource, they do not need to give the GCS uri, only the path. This just means users don't have to prefix a path with "gs://". It is implicitly understood that all resources accessed with this tool are located in GCS, with the first level of the path being a GCS bucket name.
Requirements
- Python >3.8 (3.10 is preferred)
- Poetry, install via
curl -sSL https://install.python-poetry.org | python3 -
Installation
You can install Dapla Toolbelt via pip from PyPI:
pip install dapla-toolbelt
Usage
from dapla import FileClient
from dapla import GuardianClient
import pandas as pd
# Load data using the Maskinporten Guardian client
response = GuardianClient.call_api("https://data.udir.no/api/kag", "88ace991-7871-4ccc-aaec-8fb6d78ed04e", "udir:datatilssb")
data_json = response.json()
raw_data_df = pd.DataFrame(data_json) # create pandas data frame from json
raw_data_df.head() # show first rows of data frame
FileClient.ls("bucket-name/folder") # list contents of given folder
# Save data into different formats
path_base = "bucket-name/folder/raw_data"
FileClient.save_pandas_to_json(raw_data_df, f"{path_base}.json") # generate json from data frame, and save to given path
FileClient.save_pandas_to_csv(raw_data_df, f"{path_base}.csv") # generate csv from data frame, and save to given path
FileClient.save_pandas_to_xml(raw_data_df, f"{path_base}.xml") # generate xml from data frame, and save to given path
FileClient.cat(f"{path_base}.json") # print contents of file
# Load data from different formats
# All these data frames should contain the same data:
df = FileClient.load_json_to_pandas(f"{path_base}.json") # read json from path and load into pandas data frame
df.head() # show first rows of data frame
df = FileClient.load_csv_to_pandas(f"{path_base}.csv") # read csv from path and load into pandas data frame
df.head() # show first rows of data frame
df = FileClient.load_xml_to_pandas(f"{path_base}.xml") # read xml from path and load into pandas data frame
df.head() # show first rows of data frame
Contributing
Contributions are very welcome. To learn more, see the Contributor Guide.
License
Distributed under the terms of the MIT license, Dapla Toolbelt is free and open source software.
Issues
If you encounter any problems, please file an issue along with a detailed description.
Credits
This project was generated from Statistics Norway's SSB PyPI Template.
Metadata
Release files for dapla-toolbelt 4.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| dapla_toolbelt-4.1.0.tar.gz | 17.7 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| dapla_toolbelt-4.1.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 36.2 kB
Release files / dapla_toolbelt-4.1.0.tar.gz
| Download URL | dapla_toolbelt-4.1.0.tar.gz |
|---|---|
| Size | 17.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
aaeb970a6602855faca07e3d56be68f5c4591ae968a6b8f0b15c8c66b6ee70b3
|
|
BLAKE2b-256 checksum How to use checksums |
c66b824e6d2521bb77140cdf54061c4819dd2eb8c48556afe8bbd6f7e180f9d4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/5.1.1 CPython/3.12.12
|
Release files / dapla_toolbelt-4.1.0-py3-none-any.whl
| Download URL | dapla_toolbelt-4.1.0-py3-none-any.whl |
|---|---|
| Size | 18.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
1c899faf5c6221a19e78757babbd2a8a170d1e29815dd9526c8978c7ec899c30
|
|
BLAKE2b-256 checksum How to use checksums |
06d7ed109666fabfdcfddf7d262502c149e14e093b787bfbeaa8c2e08e60761a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/5.1.1 CPython/3.12.12
|
