Skip to main content

No project description provided

Project description

Data Extract-Load-Transform for Autosink Project

🇬🇧 | 🇰🇷 | 🇨🇳

Parse the values in the data lake and load them into memory, then make it executable for the data labeler. It organizes the labeled data into a format that is easy for the feature store to store. The autosink-data-elt contains utility classes and data type definitions for reading and writing json files stored in the data lake. When the data extracted from the Raspberry Pi is stored in the data lake, it is loaded, labeled, and transformed into a form that can be stored in the feature store, so it is named ELT.

Features

  • Parse the values in the data lake and load them into memory, then organize them into a format that is easy for the data labeler to read.
  • Call the data labeler.
  • Store the labeled data in the feature store.

Environment

The environment is based on MacOS and Linux.

Makefile

The Makefile has the following functions.

make lint

  • To use the .vscode settings, install the pylint extension.
  • Override the options specified in the pyproject.toml file to lint the code with the default settings of the linter.

make format

  • The formatter uses google's yapf.
  • Override the options specified in the pyproject.toml file to format the code with the default settings of the yapf formatter.
  • To use the .vscode settings, install the yapf extension.

make test

  • The test uses unittest.
  • Supports both test_*.py and *_test.py patterns.
  • The test file must be connected to __init__.py up to the location where the test file exists.

make publish

  • Write the ~/.pypirc file as follows.
    [pypi]
    username = __token__
    password = pypi-어쩌고저쩌고 # Write your personal API token.
    
  • This command uses flit to push the package to the PyPI public registry.
  • The package uploaded with the name specified earlier as `

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

autosink_data_elt-0.2.0.tar.gz (12.9 kB view details)

Uploaded Source

Built Distribution

autosink_data_elt-0.2.0-py3-none-any.whl (10.9 kB view details)

Uploaded Python 3

File details

Details for the file autosink_data_elt-0.2.0.tar.gz.

File metadata

  • Download URL: autosink_data_elt-0.2.0.tar.gz
  • Upload date:
  • Size: 12.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: python-requests/2.31.0

File hashes

Hashes for autosink_data_elt-0.2.0.tar.gz
Algorithm Hash digest
SHA256 254d2a7ceb4c191017cdacbc19114395eb5c8b8716061381712c9e622579d33e
MD5 3e5a7ba69b9c7b8fc64ce3789e401fd0
BLAKE2b-256 f563cb66de72965312a3b55b2c7303db786187cf95a88af2a1311beef6716954

See more details on using hashes here.

Provenance

File details

Details for the file autosink_data_elt-0.2.0-py3-none-any.whl.

File metadata

File hashes

Hashes for autosink_data_elt-0.2.0-py3-none-any.whl
Algorithm Hash digest
SHA256 1de9dd147d62597b212afe9a4024364a63d8758ae1eb15e74d865da4c45a0ec0
MD5 bcbd576e345b293fdcb6590e58c21725
BLAKE2b-256 d4b5a8353fd3b087809ebb11d40a9e75c358878f88e53d280a323142f9b5227f

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page