Interact with OIP ecosystem.
Project description
rcd_dev_kit
Developed by Real Consulting Data
Description
We've developed rcd-dev-kit
to facilitate the manipulation and interaction with the OIP ecosystem.
Installation
pip install rcd-dev-kit
Modules
We've divided our functions in four main modules:
-
- Classes:
- GcloudOperator()
- MysqlOperator()
- ElasticsearchOperator()
- RedshiftOperator()
- SnowflakeOperator()
- S3Operator()
- Main Functions:
- index_json_bulk()
- index_json()
- index_json_bulk_parallel()
- send_to_redshift()
- read_from_redshift()
- send_metadata_to_redshift()
- find_tables_by_column_name()
- migrate_metadata_from_redshift()
- upload_raw_s3()
- download_raw_s3()
- upload_to_gcloud()
- download_from_gcloud()
- Classes:
-
- Classes:
- RawDataFile()
- Classes:
-
- Main Functions:
- timeit()
- debug()
- Main Functions:
-
- Classes:
- FileOperator()
- FileDownloader()
- Main Functions:
- detect_path()
- detect_all_files()
- write_df_to_json_parallel()
- download_excel()
- download_csv()
- download_pdf()
- download_zip()
- Classes:
-
- Main Functions:
- strip_all_text_column()
- check_na()
- check_duplication()
- check_quality_table_names()
- normalize_date_column()
- detect_aws_type()
- Main Functions:
-
- Main Functions:
- convert_to_snowflake_syntax()
- correct_sql_system_variables_syntax()
- Main Functions:
Pre-requirements
Since some of the functions deal with database connections(S3, Redshift, Snowflake, GCP, Elasticsearch, ...), we must
be careful to sensitive information. Thus, to use the functions correctly we must have a .env
file following
the .env.example
template.
Feedback
Any questions or suggestions? Please contact package maintainer.
python-sdk
Refer to book https://py-pkgs.org/01-introduction for best practices
Maintainers
This package is using poetry for pkg management, it must be installed locally if you are maintaining the package.
For developing and test the pkg locally, you must run poetry install
.
This git repository has an automated CI/CD process found on the git worflow: main.yml. It means that once all modifications have been made, a Pull Request to main will trigger a serie of actions:
- Install Package:
poetry install
- Run Unitary Tests:
poetry run pytest -v tests/ --cov=rcd_dev_kit --cov-report=xml
- Build Package:
poetry build
- Publish Package in PyPI:
poetry publish
- Install Package from PyPI:
pip install rcd_dev_kit
- Send a Teams message with the new available version: Git Image
toko-bifrost/ms-teams-deploy-card@master
.
Contributing
Interested in contributing? Check out the contributing guidelines. Please note that this project is released with a Code of Conduct. By contributing to this project, you agree to abide by its terms.
License
rcd_dev_kit
was created by RCD. It is licensed under the terms of the MIT license.
Credits
rcd_dev_kit
was created with cookiecutter
and the py-pkgs-cookiecutter
template.
/ Top contributors
/ Recent activity
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file rcd_dev_kit-2.5.13.tar.gz
.
File metadata
- Download URL: rcd_dev_kit-2.5.13.tar.gz
- Upload date:
- Size: 71.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.12.7
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 16396a0bbeb92b89639f01d38b8e816eb72a228d918c53d3342446beb962020b |
|
MD5 | be38686e1c067d8dd6da812393237ded |
|
BLAKE2b-256 | 927e7b19a3938971eb180ef6e04e6fae284a65746c1141e886f28f017dc50e5f |
File details
Details for the file rcd_dev_kit-2.5.13-py3-none-any.whl
.
File metadata
- Download URL: rcd_dev_kit-2.5.13-py3-none-any.whl
- Upload date:
- Size: 83.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.12.7
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4af211c6b33ce62b4644a976ab839bde9df7ae9515736edd856c92aefa4edee8 |
|
MD5 | 185068fc37606c769996dd8fc28ee08c |
|
BLAKE2b-256 | 58267c9b4330212f2cda227128a0ac2442d56f98307c4b08594970f094cd0b89 |