Dumb Log Utlity for personal use
Project description
LogNub
Loguru utility package
TODO
- Move logwrap [on top of loguru] extension out as a seperate package.
- Add Test containers for amundsen, etc..
Getting Started
- Setup SDKMAN
- Setup Java
- Setup Apache Spark
- Install Poetry
- Install Pre-commit and follow instruction in here
- Run tests locally
Setup SDKMAN
SDKMAN is a tool for managing parallel Versions of multiple Software Development Kits on any Unix based system. It provides a convenient command line interface for installing, switching, removing and listing Candidates. SDKMAN! installs smoothly on Mac OSX, Linux, WSL, Cygwin, etc... Support Bash and ZSH shells. See documentation on the SDKMAN! website.
Open your favourite terminal and enter the following:
$ curl -s https://get.sdkman.io | bash
If the environment needs tweaking for SDKMAN to be installed,
the installer will prompt you accordingly and ask you to restart.
Next, open a new terminal or enter:
$ source "$HOME/.sdkman/bin/sdkman-init.sh"
Lastly, run the following code snippet to ensure that installation succeeded:
$ sdk version
Setup Java
Install Java Now open favourite terminal and enter the following:
List the AdoptOpenJDK OpenJDK versions
$ sdk list java
Install the Java 8:
$ sdk install java 8.0.292.hs-adpt
Set Java 8 as default Java Version:
$ sdk default java 8.0.292.hs-adpt
OR
To install For Java 11
$ sdk install java 11.0.10.hs-adpt
Setup Apache Spark
Install Java Now open favourite terminal and enter the following:
List the Apache Spark versions:
$ sdk list spark
To install For Spark 3
$ sdk install spark 3.0.2
To install For Spark 3.1
$ sdk install spark 3.0.2
Install PyEnv and Python 3.8
Either install pyenv via brew or github:
brew install pyenv
Then setup in zshrc:
echo 'eval "$(pyenv init --path)"' >> ~/.zprofile
echo 'eval "$(pyenv init -)"' >> ~/.zshrc
Then Install Python 3.8:
pyenv install 3.8.11
cd allocator project directory:
# Checkout git repo of iAllocator
cd dse-iAllocator
cd customer/allocator
# Now set default python version as 3.8.11
pyenv local 3.8.11
# Verify python version
python -V
Python 3.8.11
Create VirtualENV
Create local venv inside allocator project:
cd dse-iAllocator [Repo directory]
cd customer/allocator
# Create virtualenv locally
python -m venv .venv
# Verify activate file exists
ls -G .venv/bin
# Activate virtual env python
source .venv/bin/activate
Verify virtual env and python is working and .venv is activated:
which python
# should end with {$ROOT_DIR}.venv/bin/python
which pip
# should end with {$ROOT_DIR}.venv/bin/pip
Poetry
Poetry Commands - Python package management tool
Install Poetry:
Install poetry using brew
brew install poetry
OR
Follow instructions for Linux: here
# For osx / linux / bash on windows install:
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -
Install the dep packages:
NOTE: Make sure you are connected to OpenVPN[since some are internal packages - catalog_client]
Install psycopg2 binary via PIP:
$ pip install psycopg2-binary==2.9.1
Install rest of packages via Poetry:
$ poetry install
# --tree: List the dependencies as a tree.
# --latest (-l): Show the latest version.
# --outdated (-o): Show the latest version but only for packages that are outdated.
poetry show -o
To update any package:
#$ poetry update pandas
Running Tests Locally
Take a look at tests in tests/dataquality
and tests/jobs
$ poetry run pytest
Ran 95 tests in 96.95s
Thats it, ENV is setup
NOTE: Loguru Wrap Package extracted from different internal package NOTE: It's just curated stuff, Created for personal usage.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.