ngdataenginterface 0.1.8

A library for facilitating the development of data engineering pipelines

ngdataenginterface

Installation

To install the ngdataenginterface package from PyPI, you can use pip:

pip install ngdataenginterface

installing a Specific Version

If you need to install a specific version of the package, you can specify it using the == operator:

pip install ngdataenginterface==1.0.0

Replace 1.0.0 with the desired version number.

Development Guide for ngdataenginterface

This guide provides step-by-step instructions for setting up a development environment using Docker for the ngdataenginterface project on macOS, Linux, and Windows machines.

Prerequisites

• Docker installed on your machine:
  • Docker Desktop for macOS
  • Docker Engine for Linux
  • Docker Desktop for Windows

Development Setup

1. Clone the ngdataenginterface repository with ssh:

git clone [e-mail]:ng-cash/data/ngdataenginterface.git

2. Change into the project directory:

cd ngdataenginterface

3. Build the docker image

docker build -t my-pyspark-app .

Running the Development Environment

macOS / Linux

1. Start a Docker container and mount the project directory for iterative development:

   docker run -it --rm -v "$(pwd)":/app my-pyspark-app
   

This command starts the Docker container interactively (-it flag), removes it after it exits (--rm flag), and mounts the current directory ($(pwd)) to the /app directory inside the container. You will have an interactive bash shell within the container.

2. Inside the container, you can run tests, execute scripts, and make changes to the code using your preferred editor or IDE on your host machine. The changes will be immediately available inside the container.

Windows

1. Start a Docker container and mount the project directory for iterative development:

   docker run -it --rm -v "%cd%:/app" my-pyspark-app bash
   

This command starts the Docker container interactively z(-it flag), removes it after it exits (--rm flag), and mounts the current directory (%cd%) to the /app directory inside the container. You will have an interactive bash shell within the container. Run python command to access python shell.

2. Inside the container, you can run tests, execute scripts, and make changes to the code using your preferred editor or IDE on your host machine. The changes will be immediately available inside the container.

Run the tests to verify if the configurations are correct:

ENV=dev AWS_ACCESS_KEY_ID=<aws-access-key-id> AWS_SECRET_ACCESS_KEY=<aws-secret-access-key> pytest -vv

Don't forget to change aws-access-key-id and aws-secret-access-key.

After that, run python command to access python shell, and tou are ready to go!