ngdataenginterface 0.0.4

A library for facilitating the development of data engineering pipelines

ngdataenginterface

Installation

To install the ngdataenginterface package from PyPI, you can use pip:

pip install ngdataenginterface

installing a Specific Version

If you need to install a specific version of the package, you can specify it using the == operator:

pip install ngdataenginterface==1.0.0

Replace 1.0.0 with the desired version number.

Development

Instaling Java and PySpark

PySpark uses Java underlying hence you need to have Java on your Mac. Since Java is a third party, you can install it using the Homebrew command brew. Since Oracle Java is not open source anymore, I am using the OpenJDK version 11. Run the below command in the terminal to install it.

brew install openjdk@11

Creating a Virtual Environment

It's recommended to create a virtual environment before working on the ngdataenginterface package. You can use venv to create a virtual environment:

python -m venv myenv

Activate the virtual environment:

• On Windows

.\env\Scripts\activate

• On macOS and Linux

source myenv/bin/activate

Installing Locally

To install the ngdataenginterface package locally for development, navigate to the package directory and run:

pip install -e .

The -e flag stands for "editable," which creates a symbolic link to the package directory. This allows you to make changes to the code and see the effects without reinstalling the package.