Skip to main content

OS-Climate Python tools to assist with standardized data ingestion workflows

Project description

osc-ingest-tools

python tools to assist with standardized data ingestion workflows

Installation, Usage, and Release Management

Install from PyPi

pip install osc-ingest-tools

Examples

>>> from osc_ingest_trino import *

>>> import pandas as pd

>>> data = [['tom', 10], ['nick', 15], ['juli', 14]]

>>> df = pd.DataFrame(data, columns = ['First Name', 'Age In Years']).convert_dtypes()

>>> df
  First Name  Age In Years
0        tom            10
1       nick            15
2       juli            14

>>> enforce_sql_column_names(df)
  first_name  age_in_years
0        tom            10
1       nick            15
2       juli            14

>>> enforce_sql_column_names(df, inplace=True)

>>> df
  first_name  age_in_years
0        tom            10
1       nick            15
2       juli            14

>>> df.info(verbose=True)
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3 entries, 0 to 2
Data columns (total 2 columns):
 #   Column        Non-Null Count  Dtype
---  ------        --------------  -----
 0   first_name    3 non-null      string
 1   age_in_years  3 non-null      Int64
dtypes: Int64(1), string(1)
memory usage: 179.0 bytes

>>> p = create_table_schema_pairs(df)

>>> print(p)
    first_name varchar,
    age_in_years bigint

>>>

Adding custom type mappings to create_table_schema_pairs

>>> df = pd.DataFrame(data, columns = ['First Name', 'Age In Years'])

>>> enforce_sql_column_names(df, inplace=True)

>>> df.info(verbose=True)
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3 entries, 0 to 2
Data columns (total 2 columns):
 #   Column        Non-Null Count  Dtype
---  ------        --------------  -----
 0   first_name    3 non-null      object
 1   age_in_years  3 non-null      int64
dtypes: int64(1), object(1)
memory usage: 176.0+ bytes

>>> p = create_table_schema_pairs(df, typemap={'object':'varchar'})

>>> print(p)
    first_name varchar,
    age_in_years bigint

>>>

Development

Patches may be contributed via pull requests to https://github.com/os-climate/osc-ingest-tools.

All changes must pass the automated test suite, along with various static checks.

Black code style and isort import ordering are enforced.

Enabling automatic formatting via pre-commit is recommended:

pip install black isort pre-commit
pre-commit install

To ensure compliance with static check tools, developers may wish to run;

pip install black isort
# auto-sort imports
isort .
# auto-format code
black .

Code can then be tested using tox.

# run static checks and tests
tox
# run only tests
tox -e py3
# run only static checks
tox -e static
# run tests and produce a code coverage report
tox -e cov

Releasing

To release a new version of this library, authorized developers should;

  • Prepare a signed release commit updating version in setup.py
  • Tag the commit using Semantic Versioning prepended with "v"
  • Push the tag

E.g.,

git commit -sm "Release v0.3.4"
git tag v0.3.4
git push --follow-tags

A Github workflow will then automatically release the version to PyPI.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

osc_ingest_tools-0.5.3.tar.gz (16.5 kB view details)

Uploaded Source

Built Distribution

osc_ingest_tools-0.5.3-py3-none-any.whl (16.6 kB view details)

Uploaded Python 3

File details

Details for the file osc_ingest_tools-0.5.3.tar.gz.

File metadata

  • Download URL: osc_ingest_tools-0.5.3.tar.gz
  • Upload date:
  • Size: 16.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/5.0.0 CPython/3.12.3

File hashes

Hashes for osc_ingest_tools-0.5.3.tar.gz
Algorithm Hash digest
SHA256 8eda6ca222c29937355abac7084bf3605582cad042068c06b6639e22edeb1083
MD5 5af96b2f822e2d42d937bde171c6227c
BLAKE2b-256 a51ddfb3f22104836d3c37cc7dee94e962e5d6f2ddff7bba56d0610dec517621

See more details on using hashes here.

File details

Details for the file osc_ingest_tools-0.5.3-py3-none-any.whl.

File metadata

File hashes

Hashes for osc_ingest_tools-0.5.3-py3-none-any.whl
Algorithm Hash digest
SHA256 00714eed7c43bfc07d938ac931b58535406cfbeb45e98bebc861287d7504f4ab
MD5 8e8b960df79b7c8a5c6a415cd7dbdf8a
BLAKE2b-256 b5bfe00bd07ffbaf04c8172ea15f6525a551ae6409abad3aba4cdf3932a085aa

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page