OS-Climate Python tools to assist with standardized data ingestion workflows
Project description
osc-ingest-tools
python tools to assist with standardized data ingestion workflows
Installation, Usage, and Release Management
Install from PyPi
pip install osc-ingest-tools
Examples
>>> from osc_ingest_trino import *
>>> import pandas as pd
>>> data = [['tom', 10], ['nick', 15], ['juli', 14]]
>>> df = pd.DataFrame(data, columns = ['First Name', 'Age In Years']).convert_dtypes()
>>> df
First Name Age In Years
0 tom 10
1 nick 15
2 juli 14
>>> enforce_sql_column_names(df)
first_name age_in_years
0 tom 10
1 nick 15
2 juli 14
>>> enforce_sql_column_names(df, inplace=True)
>>> df
first_name age_in_years
0 tom 10
1 nick 15
2 juli 14
>>> df.info(verbose=True)
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3 entries, 0 to 2
Data columns (total 2 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 first_name 3 non-null string
1 age_in_years 3 non-null Int64
dtypes: Int64(1), string(1)
memory usage: 179.0 bytes
>>> p = create_table_schema_pairs(df)
>>> print(p)
first_name varchar,
age_in_years bigint
>>>
Adding custom type mappings to create_table_schema_pairs
>>> df = pd.DataFrame(data, columns = ['First Name', 'Age In Years'])
>>> enforce_sql_column_names(df, inplace=True)
>>> df.info(verbose=True)
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3 entries, 0 to 2
Data columns (total 2 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 first_name 3 non-null object
1 age_in_years 3 non-null int64
dtypes: int64(1), object(1)
memory usage: 176.0+ bytes
>>> p = create_table_schema_pairs(df, typemap={'object':'varchar'})
>>> print(p)
first_name varchar,
age_in_years bigint
>>>
Development
Patches may be contributed via pull requests to https://github.com/os-climate/osc-ingest-tools.
All changes must pass the automated test suite, along with various static checks.
Black code style and isort import ordering are enforced.
Enabling automatic formatting via pre-commit is recommended:
pip install black isort pre-commit
pre-commit install
To ensure compliance with static check tools, developers may wish to run;
pip install black isort
# auto-sort imports
isort .
# auto-format code
black .
Code can then be tested using tox.
# run static checks and tests
tox
# run only tests
tox -e py3
# run only static checks
tox -e static
# run tests and produce a code coverage report
tox -e cov
Releasing
To release a new version of this library, authorized developers should;
- Prepare a signed release commit updating
version
in setup.py - Tag the commit using Semantic Versioning prepended with "v"
- Push the tag
E.g.,
git commit -sm "Release v0.3.4"
git tag v0.3.4
git push --follow-tags
A Github workflow will then automatically release the version to PyPI.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file osc_ingest_tools-0.5.3.tar.gz
.
File metadata
- Download URL: osc_ingest_tools-0.5.3.tar.gz
- Upload date:
- Size: 16.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/5.0.0 CPython/3.12.3
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8eda6ca222c29937355abac7084bf3605582cad042068c06b6639e22edeb1083 |
|
MD5 | 5af96b2f822e2d42d937bde171c6227c |
|
BLAKE2b-256 | a51ddfb3f22104836d3c37cc7dee94e962e5d6f2ddff7bba56d0610dec517621 |
File details
Details for the file osc_ingest_tools-0.5.3-py3-none-any.whl
.
File metadata
- Download URL: osc_ingest_tools-0.5.3-py3-none-any.whl
- Upload date:
- Size: 16.6 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/5.0.0 CPython/3.12.3
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 00714eed7c43bfc07d938ac931b58535406cfbeb45e98bebc861287d7504f4ab |
|
MD5 | 8e8b960df79b7c8a5c6a415cd7dbdf8a |
|
BLAKE2b-256 | b5bfe00bd07ffbaf04c8172ea15f6525a551ae6409abad3aba4cdf3932a085aa |