Skip to main content

Dumb Log Utlity for personal use

Project description

LogNub

BuildAndTest PreCommitChecks CodeQL codecov

Loguru utility package

TODO

  1. Move logwrap [on top of loguru] extension out as a seperate package.
  2. Add Test containers for amundsen, etc..

Getting Started

  1. Setup SDKMAN
  2. Setup Java
  3. Setup Apache Spark
  4. Install Poetry
  5. Install Pre-commit and follow instruction in here
  6. Run tests locally

Setup SDKMAN

SDKMAN is a tool for managing parallel Versions of multiple Software Development Kits on any Unix based system. It provides a convenient command line interface for installing, switching, removing and listing Candidates. SDKMAN! installs smoothly on Mac OSX, Linux, WSL, Cygwin, etc... Support Bash and ZSH shells. See documentation on the SDKMAN! website.

Open your favourite terminal and enter the following:

$ curl -s https://get.sdkman.io | bash
If the environment needs tweaking for SDKMAN to be installed,
the installer will prompt you accordingly and ask you to restart.

Next, open a new terminal or enter:

$ source "$HOME/.sdkman/bin/sdkman-init.sh"

Lastly, run the following code snippet to ensure that installation succeeded:

$ sdk version

Setup Java

Install Java Now open favourite terminal and enter the following:

List the AdoptOpenJDK OpenJDK versions
$ sdk list java

Install the Java 8:
$ sdk install java 8.0.292.hs-adpt

Set Java 8 as default Java Version:
$ sdk default java 8.0.292.hs-adpt

OR 

To install For Java 11
$ sdk install java 11.0.10.hs-adpt

Setup Apache Spark

Install Java Now open favourite terminal and enter the following:

List the Apache Spark versions:
$ sdk list spark

To install For Spark 3
$ sdk install spark 3.0.2

To install For Spark 3.1
$ sdk install spark 3.0.2

Install PyEnv and Python 3.8

Either install pyenv via brew or github:

brew install pyenv

Then setup in zshrc:
echo 'eval "$(pyenv init --path)"' >> ~/.zprofile

echo 'eval "$(pyenv init -)"' >> ~/.zshrc

Then Install Python 3.8:

pyenv install 3.8.11

cd allocator project directory:

# Checkout git repo of iAllocator
cd dse-iAllocator 
cd customer/allocator
# Now set default python version as 3.8.11
pyenv local 3.8.11
# Verify python version
python -V
Python 3.8.11

Create VirtualENV

Create local venv inside allocator project:

cd dse-iAllocator [Repo directory] 
cd customer/allocator
# Create virtualenv locally 
python -m venv .venv
# Verify activate file exists 
ls -G .venv/bin
# Activate virtual env python
source .venv/bin/activate

Verify virtual env and python is working and .venv is activated:

which python
# should end with {$ROOT_DIR}.venv/bin/python
which pip
# should end with {$ROOT_DIR}.venv/bin/pip

Poetry

Poetry Commands - Python package management tool

Install Poetry:

Install poetry using brew

brew install poetry

OR

Follow instructions for Linux: here

# For osx / linux / bash on windows install:
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -

Install the dep packages:

NOTE: Make sure you are connected to OpenVPN[since some are internal packages - catalog_client]

Install psycopg2 binary via PIP:
$ pip install psycopg2-binary==2.9.1

Install rest of packages via Poetry:

$ poetry install

# --tree: List the dependencies as a tree.
# --latest (-l): Show the latest version.
# --outdated (-o): Show the latest version but only for packages that are outdated.
poetry show -o

To update any package:
#$ poetry update pandas

Running Tests Locally

Take a look at tests in tests/dataquality and tests/jobs

$ poetry run pytest
Ran 95 tests in 96.95s

Thats it, ENV is setup

NOTE: Loguru Wrap Package extracted from different internal package NOTE: It's just curated stuff, Created for personal usage.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

lognub-0.2.1.tar.gz (13.0 kB view details)

Uploaded Source

Built Distribution

lognub-0.2.1-py3-none-any.whl (14.2 kB view details)

Uploaded Python 3

File details

Details for the file lognub-0.2.1.tar.gz.

File metadata

  • Download URL: lognub-0.2.1.tar.gz
  • Upload date:
  • Size: 13.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.10.11

File hashes

Hashes for lognub-0.2.1.tar.gz
Algorithm Hash digest
SHA256 26c3c08ced0fcf51f1879a19647e9e4199a1649f722ccfcce0e0541f15dd4a5a
MD5 d54bb5788cd2146f38f85e247e1ef98c
BLAKE2b-256 57af9f27b2488a4a925ed0eec017f21dcf36143ebdedbad79afaa5ec522dc897

See more details on using hashes here.

File details

Details for the file lognub-0.2.1-py3-none-any.whl.

File metadata

  • Download URL: lognub-0.2.1-py3-none-any.whl
  • Upload date:
  • Size: 14.2 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.10.11

File hashes

Hashes for lognub-0.2.1-py3-none-any.whl
Algorithm Hash digest
SHA256 3d11c2f4810216b6a92ed3e39209739cb6ae3da74dc45c3f35a0bcc715bf1d97
MD5 5286ca8af06ca9ccd63fb0e894fb8797
BLAKE2b-256 a7ea702ff14e71d4e894edbffbe3763ccf2609cca80cdd34d387a09554aadb74

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page