logsparsersubsystem 1.0.4

logs parsing subsystem library for airflow

LogsParser

Common log parser and analyzer

Project structure

How to dev

poetry shell
poetry install

---

├── debug.py
├── docs
│   ├── context diagram.drawio
│   └── uml.drawio
├── logs_parser
│   ├── analyzer - analyzer for logs
│   │   ├── analyzer.py
│   │   ├── data - data for IP2Location Lib
│   │   │   ├── IP2LOCATION-LITE-DB11.BIN
│   │   │   ├── LICENSE_LITE.TXT
│   │   │   └── README_LITE.TXT
│   │   └── plots.py - plots (deprecated)
│   ├── parser (multithread parser)
│   │   ├── filter.py - filter decorators
│   │   ├── logs_reader.py - logs reader
│   │   ├── log_structure.py - data classes
│   │   ├── logs_writer.py - logs writers
│   │   └── parser.py - parser
│   └── python_callables.py - callables for Airflow
├── notebooks - Jupyter notebooks,some experiment
│   ├── experiment.ipynb
│   └── gorod_09102021_231021.csv
├── poetry.lock
├── pyproject.toml
├── README.md
└── scripts - utils
    ├── split_logs.py
    └── unite_logs.py

---

Arhitecture contex diagram

UML diagram

How to public to Pypi

First you need create api on pypi

poetry config pypi-token.pypi $(cat .token)
poetry publish --build