genov 0.0.6

Genovation toolbox

The Genovation python toolbox

This python tool-box is made to support Genovation associates in their day-to-day work.

INSTALL

Pre-requisites

• Windows PowerShell (Terminal on MS Windows)

  • Console to run genov tool box, and its commands
  • How-to from Microsoft: https://learn.microsoft.com/en-us/powershell/scripting/install/installing-powershell-on-windows

• Python:

  • genov is tested with Python 3.13
  • As of EoY 2024, Python 3.13.0 is available at: https://www.python.org/downloads/
  • Note: starting with Python 3.4, pip is by default included

Once installed, you can check in your terminal:

foo@bar:~$ python3 --version
           >> Python 3.13.0
foo@bar:~$ pip3 --version
           >> pip 24.3.1 from /Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/pip (python 3.13)

Notes:

• On your system, you might have to replace the python3 command with python
• To check where your python is installed: which python3: /Library/Frameworks/Python.framework/Versions/3.13/bin/python3

Install Genov

• Type in your Terminal: pip3 install genov

Versions

• 0.0.1, as of 29-Nov-2024: framework initialized
• 0.0.2, as of 3-Dec-2024: get issues from jira, and persist in an Excel spreadsheet
• 0.0.4, as of 2-Jan-2025: framework enriched with test coverage, lint, typer documentation and sphinx documentation
• 0.0.5, as of 18-Mar-2025: DpDatePredictability, along with typer commands (CAUTION: poor performance...)
• 0.0.6, as of 22-Mar-2026: DpDataPredictability fully reengineered

Contribute

Dependencies

To run

Dependencies	Description
com-enovation	The seed toolbox that we use to initialize this toolbox. To decommission as commands are being re-instantiated here...
typer	Library for building CLI applications, based on Click
tomlkit	Library for manipulating configuration file
pydash	Library for manipulating dictionaries with path...
jsonschema	Library for validating json schema
jsonata-python	Library for exploring dictionaries and json
openpyxl	Library for reading and writing excel files

To contribute

Dependencies	Description
hatch	Extensible, standards compliant build backend.
pytest	To run tests (natively imported by hatch).
pytest-cov	To measure test coverage (natively imported by hatch).
sphinx	To document project
sphinx-autodoc-typehints	To use Python 3 annotations for documenting acceptable argument types and return value types of functions
sphinx_rtd_theme	The 'read the docs' theme

Steps to publish changes/ enhancements to genov

Code formatting and linting

• Run hatch fmt
  • Ensure you have no error
  • Message of the kind should be returned: Found 2 errors (2 fixed, 0 remaining)
• Notes:
  • Linting analyzes how the code runs and detects errors
  • Formatting restructures how code appears.

Test coverage

• Run GENOV_TEST_LEVEL=FULL hatch test --cover --strict
  • Ensure you have no error, and that you effectively have 80% coverage
• Notes:
  • Target is to have 80% of the code covered with tests
  • No publication allowed in case an average of less than 80%
  • Environment variable GENOV_TEST_LEVEL can take value FULL or DEFAULT, and is then reflected in test/conftest.py --> e_test_level.

Documentation

Readme.md

• Run hatch run docs:md
  • Message of the kind should be returned: Docs saved to: GENOV.md
• Notes: the command line above executes the following steps
  • The typer command to generate docs for our package that we access through our README.md
  • Requires an environment where the distribution is installed
    • rm -rf dist;hatch build;pip3 install dist/genov-*.tar.gz
  • Run the typer command to document:
    • Step 1, we call the command to produce the document: typer genov.genov utils docs --output GENOV.md.raw --name genov
    • Step 2, we reformat quotes, so they are properly handled: awk '{gsub(/"/, ""); print}' GENOV.md.raw > GENOV.md;
  • Modify the README.md file to append the typer documentation:
    • Step 1, we rename the file: mv README.md README.md.previous
    • Step 2, we discard the tail of the readme file (pattern modified to Otomated documentation belowso the script is not confused...): awk '!(f>3);/Otomated documentation below/{f=1}; {if (f) {f=f+1}};' README.md.previous > README.md.tmp
    • Step 3, we concatenate the latest documentation: cat README.md.tmp GENOV.md > README.md
    • Step 4, we cleanse the tmp version: rm README.md.tmp
  • Final command is: rm -rf dist;hatch build;pip3 install dist/genov-*.tar.gz;typer genov.genov utils docs --output GENOV.md.raw --name genov;awk '{{gsub(/"/, \"\\\"\"); print}}' GENOV.md.raw > GENOV.md;mv README.md README.md.previous; awk '!(f>3);/Otomated documentation below/{{f=1}}; {{if (f) {{f=f+1}}}};' README.md.previous > README.md.tmp;cat README.md.tmp GENOV.md > README.md;rm README.md.tmp

Sphinx documentation

• Run hatch run docs:sphinx
  • Message of the kind should be returned: The HTML pages are in [./docs/]_build/html.
• Note:
  • To auto-document the entire application, we use sphinx
  • The command is make html from the sub-directory ./docs
  • To document class constants, use the following

    #: The description of my constant goes there, after "#:"
    MY_CONSTANT: str = "Value for my constant"
    

Publish

• Update the version
  • In pyproject.toml
  • In README.md (along with a light description of the updates)
• Run rm -rf dist/*;hatch build;hatch publish -r test (To build and publish in test)
  • Directory dist is generated (with distribution files)
  • Message of the kind should be returned: https://test.pypi.org/project/genov/[...latest version...]/
• Run hatch publish (Once verified in test, to publish in production)
  • Message of the kind should be returned: https://pypi.org/project/genov/[...latest version...]/
• Notes:
  • Python equivalent command to publish: python3 -m twine upload --repository pypi dist/*
  • Package viewable at pypi

Pycharm configuration

• Unit test configuration, from menu Run > Edit Configurations...
  • Configuration > Target > Script path: ~/PycharmProjects/com_enovation.murex/tests
  • Configuration > Working directory: ~/PycharmProjects/com_enovation.murex/
  • Configuration > Add content roots to PYTHONPATH: checked
  • Configuration > Add source roots to PYTHONPATH: checked
  • Configuration > Additional Arguments: --assert=plain: to get assert messages when executing tests

Python Cheat sheet

• To upgrade python:

  • The version used as the interpreter in PyCharm project, with simlinks under PycharmProjects/genov/venv/bin: brew upgrade python
    • Stored under /opt/homebrew/bin/python3
  • The version used by MacOsX: re-download/ re-install from python.org
    • Stored under cd /usr/local/bin --> /Library/Frameworks/Python.framework/Versions/3.13/bin
  • To install python: brew install python
    • See https://brew.sh/ to install brew`

• To get emojis that can be printed by rich.print(): run python -m rich.emoji in console

• Check we have the latest versions:

  • pip: python -m pip install --upgrade pip
  • build to generate the distribution: python -m pip install --upgrade build

• Update packages using pip

  • Check all packages are fine: pip check
  • List all packages outdated: pip list --outdated
  • Update all packages outdated:
    • On Mac: pip --disable-pip-version-check list --outdated --format=json | python -c "import json, sys; print('\n'.join([x['name'] for x in json.load(sys.stdin)]))" | xargs -n1 pip install -U
    • On Windows: pip freeze | %{$_.split('==')[0]} | %{pip install --upgrade $_}

• A simple example package. You can use Github-flavored Markdown to write your content.

---

WARNING: do not modify below, as automatically override by script `hatch run docs:md!!!

Automated documentation below

genov

Genov tool box, the application with all the commands you need in your day-to-day work at Genovation.

Use the VERBOSE parameter to set the level of logs you need, and let you guide by the HELP.

Usage:

$ genov [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...

Options:

• -v, --verbose / --no-verbose: Level of logging verbosity: INFO (--verbose), WARNING (default) or ERROR (--no-verbose). [default: (WARNING)]
• --install-completion: Install completion for the current shell.
• --show-completion: Show completion for the current shell, to copy it or customize the installation.
• --help: Show this message and exit.

Commands:

• welcome: The command welcomes to the Genovation...
• jr-gt-iss: The command fetches issues from a Jira...
• df-to-stdout: The command prints into console the...
• df-to-xlsx: The command persists the DataFrame aliased...
• df-load-xlsx: The command loads FILE from the file...
• dict-to-stdout: The command prints into console the...
• dict-to-json: The command persists the dictionary...
• cfg-to-stdout: The command prints into console the...
• dp: The command instantiates an instance of...
• dp-asof
• dp-compute: The command computes the Date...
• dp-decompose: The command decomposes the Date...
• dp-slide: The command slides the Date Predictability...
• dp-describe: The command describes the Date...

genov welcome

The command welcomes to the Genovation toolbox! This command will greet NAME, and return welcome message. It experiments the rich library that can be used across other commands.

Usage:

$ genov welcome [OPTIONS] NAME

Arguments:

• NAME: [required]

Options:

• --help: Show this message and exit.

genov jr-gt-iss

The command fetches issues from a Jira PROJECT within a given SITE using a predefined TOKEN belonging to an ACCOUNT. The result is transformed through a MAP, returned depending on AS_DF as a DataFrame or a dict, and stored in context as ALIAS.

To connect to the underlying Atlassian Rest API, the following parameters are required from the configuration file under ~/.genov/genov.toml to retrieve the following parameters:

• SITE: a site contains instances of Atlassian products under an organization. An organization has one or more sites, each site can only have one instance of each product -> path in configuration file: the site from atlassian.sites where label is SITE
• PROJECT: a project is a collection of issues (stories, bugs, tasks, etc.), used to represent the development work for a product, project, or service in Jira -> path in configuration file: the project from atlassian.sites.jira.projects where label is PROJECT
• Resource: the REST resources available in Jira Cloud -> path in configuration file: the resource from atlassian.sites.jira.resources where label is 'GET /rest/api/3/search/jql'
• MAP: used to transform complex and proprietary responses from Atlassian Rest APIs in data we can process, such as DataFrame, Dictionaries, etc. -> path in configuration file: the map from atlassian.sites.jira.resources.maps where label is MAP
• ACCOUNT: an Atlassian account is an online Atlassian identity that exists independently of the Atlassian products that is used. An account is required to log in to any Atlassian products, such as Jira or Confluence. An Atlassian account is like a Google account. When you log into your Gmail account, you can also log in to YouTube, Google Docs, etc. -> path in configuration file: the account from atlassian.accounts where label is ACCOUNT
• TOKEN: API tokens are used to authenticate users when making calls to Atlassian product APIs -> path in configuration file: the token from atlassian.accounts where label is TOKEN.

Illustration of the expected configuration in the ~/.genov/genov.toml file

.. Code-block:: text

{
    "atlassian": {
        "accounts": [
            {
                "label": "pro",
                "email": "[e-mail]",
                "tokens": [
                    {
                        "label": "GSculpt",
                        "token": "whatever"
                    }
                ]
            }
        ],
        "sites": [
            {
                "label": "genovation",
                "jira": {
                    "projects": [
                        {
                            "label": "GENOVATION",
                            "key": "GENO"
                        }
                    ],
                "resources": [
                    {
                        "label": "GET /rest/api/3/search/jql",
                        "url": "https://genovation.atlassian.net/rest/api/3/search",
                        "maps": [
                            {
                                "label": "flatList",
                                "map": [
                                    {
                                        "dict.path": "id",
                                        "df.column": "id",
                                        "df.type": "int"
                                    }
                                ]
                            }
                        ]
                    }
                ]
            }
        ]
    }
}

Usage:

$ genov jr-gt-iss [OPTIONS]

Options:

• -s, --site site: Atlassian site to consider. If not set, defaulted to first configured site. [default: (If not provided, defaulted with the first site configured in the configuration file)]
• -p, --project project: Atlassian jira project to consider. If not set, defaulted to first configured project. [default: (If not provided, defaulted with the first project configured in the configuration file)]
• -m, --map map: Map to transform Atlassian responses. If not set, defaulted to first configured map. [default: (If not provided, defaulted with the first map configured in the configuration file)]
• --account account: Atlassian account to consider. If not set, defaulted to first configured account. [default: (If not provided, defaulted with the first account configured in the configuration file)]
• -t, --token token: The Atlassian token to connect to the REST APIs. [default: (If not provided, defaulted with the first token configured in the configuration file)]
• --as-df / --as-dict: Either get the result as a DataFrame or as a dict. [default: (As DataFrame)]
• -a, --alias alias: The alias for the issues stored in context. [default: (issues)]
• --help: Show this message and exit.

genov df-to-stdout

The command prints into console the dataframe instance that is stored in context as ALIAS.

Usage:

$ genov df-to-stdout [OPTIONS] alias

Arguments:

• alias: The alias for the dataframe stored in context to be printed. [required]

Options:

• --help: Show this message and exit.

genov df-to-xlsx

The command persists the DataFrame aliased ALIAS in the file system as FILE.

Usage:

$ genov df-to-xlsx [OPTIONS] alias file

Arguments:

• alias: The alias for the dataframe stored in context to be printed. [required]
• file: The file to export the dataframe. [required]

Options:

• --help: Show this message and exit.

genov df-load-xlsx

The command loads FILE from the file system in a DataFrame that is staged in context under alias DF ALIAS. In case only a subset of columns is to be loaded, these can be listed under COLUMNS, so useless columns are discarded. KWARGS can be used to provide any technical parameters to the underlying function "pandas.read_excel".

Usage:

$ genov df-load-xlsx [OPTIONS] file

Arguments:

• file: The MS Excel file to import as a dataframe. [required]

Options:

• -o, --df_output Df Alias: Alias for the loaded Date Frame to be persisted in context. [default: (If not provided, defaulted to '_df'.)]
• -c, --columns columns: Columns labels to load, others will be discarded. [default: (By default, all columns are loaded.)]
• -k, --kwargs kwargs: Parameters for the underlying function pandas.read_excel. [default: (By default, no parameter provided.)]
• --help: Show this message and exit.

genov dict-to-stdout

The command prints into console the dictionary instance that is stored in context as ALIAS.

Usage:

$ genov dict-to-stdout [OPTIONS] alias

Arguments:

• alias: The alias for the json stored in context to be printed. [required]

Options:

• --help: Show this message and exit.

genov dict-to-json

The command persists the dictionary instance that is stored in context as ALIAS in the file system as FILE.

Usage:

$ genov dict-to-json [OPTIONS] alias file

Arguments:

• alias: The alias for the JSON stored in context to be printed. [required]
• file: The file to export the dict. [required]

Options:

• --help: Show this message and exit.

genov cfg-to-stdout

The command prints into console the content of the configuration file stored under ~/.genov/genov.toml.

Usage:

$ genov cfg-to-stdout [OPTIONS]

Options:

• --help: Show this message and exit.

genov dp

The command instantiates an instance of Date Predictability that is then staged in context under DP ALIAS:

• It is initialized with the backlog of work items retrieved from context as a DataFrame aliased DF ALIAS.
• The columns' labels from the work items DataFrame are configured: KEY, LABEL, MEASURE, STATUS.

Usage:

$ genov dp [OPTIONS] df_alias

Arguments:

• df_alias: The alias for the dataframe stored in context, used to compute predictability upon. [required]

Options:

• -a, --as dp_alias: Alias for the Date Predictability instance that is persisted in context. The Date Predictability instance is later used to execute commands such as COMPUTE, DECOMPOSE or SLIDE. [default: (If not provided, defaulted to _dp)]
• -k, --key key: Column in the dataframe that contains the KEYs. [default: (If not provided, defaulted to 'key'.)]
• -l, --label label: Column in the dataframe that contains the LABELs. [default: (If not provided, defaulted to 'label'.)]
• -m, --measure measure: Column(s) in the dataframe that contain (s) the MEASUREs, either SPOT (e.g. eta) or RANGE (e.g. '("from", "to")', surrounded with simple quotes). [default: (If not provided, defaulted to 'measure', aka SPOT.)]
• -s, --status status: Column in the dataframe that contains the STATUSes. [default: (If not provided, defaulted to 'status'.)]
• --help: Show this message and exit.

genov dp-asof

Usage:

$ genov dp-asof [OPTIONS] int_asof

Arguments:

• int_asof: The LABEL to truncate posterior observation. [required]

Options:

• -i, --dp_instance Dp Alias: The Date Predictability instance to use. [default: (If not provided, defaulted to '_dp'.)]
• --help: Show this message and exit.

genov dp-compute

The command computes the Date Predictability by key, from the Date Predictability instance DP ALIAS that is retrieved in context. Eventually, computed Date Predictability is persisted in context as a DataFrame aliased DF ALIAS.

Usage:

$ genov dp-compute [OPTIONS]

Options:

• -i, --dp_instance dp_alias: The Date Predictability instance to use. [default: (If not provided, defaulted to '_dp'.)]
• -o, --df_output Df Alias: Alias for the computed Date Frame persisted in context. [default: (If not provided, defaulted to '_df'.)]
• --help: Show this message and exit.

genov dp-decompose

The command decomposes the Date Predictability for a given KEY, from the Date Predictability instance DP ALIAS that is retrieved in context. Eventually, decomposed Date Predictability is persisted in context as a DataFrame aliased DF ALIAS.

Usage:

$ genov dp-decompose [OPTIONS]

Options:

• --str-key TEXT
• -i, --dp_instance Dp Alias: The Date Predictability instance to use. [default: (If not provided, defaulted to '_dp'.)]
• -o, --df_output Df Alias: Alias for the computed Date Frame persisted in context. [default: (If not provided, defaulted to '_df'.)]
• --help: Show this message and exit.

genov dp-slide

The command slides the Date Predictability for a given KEY, from the Date Predictability instance DP ALIAS that is retrieved in context. Eventually, slid Date Predictability is persisted in context as a DataFrame aliased DF ALIAS.

Usage:

$ genov dp-slide [OPTIONS]

Options:

• --str-key TEXT
• -i, --dp_instance dp_lias: The Date Predictability instance to use. [default: (If not provided, defaulted to '_dp'.)]
• -o, --df_output df_alias: Alias for the computed Date Frame persisted in context. [default: (If not provided, defaulted to '_df'.)]
• --help: Show this message and exit.

genov dp-describe

The command describes the Date Predictability data, from the Date Predictability instance DP ALIAS that is retrieved in context. Eventually, Date Predictability statistics are persisted in context as a dict aliased DICT_ALIAS.

Usage:

$ genov dp-describe [OPTIONS]

Options:

• -i, --dp_instance Dp Alias: The Date Predictability instance to use. [default: (If not provided, defaulted to '_dp'.)]
• -o, --dict_output dict_alias: Alias for the computed statistics persisted in context. [default: (If not provided, defaulted to '_dict'.)]
• --help: Show this message and exit.