srcopsmetrics 2.9.0

No project description provided

Overview

MI project collects data from GitHub repositories. You can use it to either collect data stored locally or within Amazon’s S3 cloud. For personal usage, checkout <Usage> section.

Together with mi-scheduler, we provide automated data extraction pipeline for data minig of requested repositories and organizations. This pipeline can be scheduled customly, e.g. to run daily, weekly, and so on.

Data extraction request

To request data extraction for repository or organization, create Data Extraction Issue in MI-Scheduler repository. Use this link TODO

Data extraction Pipeline (diagram)

MI pipeline is simple to understand, see diagram below

                  +---------+
                  |ConfigMap|
                  +----+----+
                       |
            +--+-------+--------+--+
            |  |                |  |
            |  |  mi-scheduler  |  |
            |  |                |  |
            +------+---+---+-------+
                |   |   |   |    |
                |   |   |   |    |
                |   |   |   |    |
                | Argo Workflows |
                |   |   |   |    |
                |   |   |   |    |
+---------------v---v---v---v----v------------------+                                          +--------------------        +--------------------+
|                                                   |                                          |   Visualization   |        |   Recommendation   |
|  +---------+  +---------+            +---------+  |                                          +-------------------+        +--------------------+
|  |thoth/   |  |  AICoE  |            | your    |  |                                          |   Project Health  |        |   thoth            |
|  |  station|  |         |            |     org |  |                                          |    (dashboard)    |        |                    |
|  +---------+  +---------+            +---------+  |                                          |                   |        |                    |
|  |solver   |  |...      |            |your     |  |                                          +---------+---------+        +----------+---------+
|  |         |  |         |            |   repos |  |           thoth-station/mi                         ^                             ^
|  |amun     |  |...      | X X X X X  |         |  |     (Meta-information Indicators)                  |                             |
|  |         |  |         |            |         |  |                                                    +-------------+---------------+
|  |adviser  |  |...      |            |         |  |                                                                  |
|  |         |  |         |            |         |  |                                                                  |
|  |....     |  |...      |            |         |  |                                                +-----------------+-------------------+
|  |         |  |         |            |         |  |                                                |                                     |
|  +---------+  +---------+            +---------+  |                                                |       Knowledge Processsing         |
|                                                   |                                                |                                     |
+-----------------------+---------------------------+                                                +-----------------+-------------------+
GitHub repositories   |                                                                                              ^
                        |                 +--------------------------------------------------------+                   |
                        |                 |                                                        |                   |
                        |                 |      Entities Analysis   +------->      Knowledge      |                   |
                        +---------------->-+                                                      +--------------------+
                                          +---------+----------------+----------+------------------+
                                          |  Issues |  Pull Requests |  Readmes |  etc...........  |
                                          |         |                |          |                  |
                                          +---------+----------------+----------+------------------+

What can MI extract from GitHub?

MI analyses entities specified on the srcopsmetrics/entities page Entity is essentialy a repository metadata that is being inspected (e.g. Issue or Pull Request), from which specified features are extracted and are stored to dataframe.

MI is essentialy wrapped around PyGitHub module to provide careless data extraction with API rate limit handling and data updating.

Install

pip

MI is available through PyPI, so you can do

pip install srcopsmetrics

git

Alternatively, you can install srcopsmetrics by cloning repository

git clone https://github.com/thoth-station/mi.git

cd mi

pipenv install --dev

Usage

Setup

To store data locally, use -l when calling CLI or set is_local=True when using MI as a module.

By default MI will try to store the data on Ceph. In order to store on Ceph you need to provide the following env variables:

• S3_ENDPOINT_URL Ceph Host name

• CEPH_BUCKET Ceph Bucket name

• CEPH_BUCKET_PREFIX Ceph Prefix

• CEPH_KEY_ID Ceph Key ID

• CEPH_SECRET_KEY Ceph Secret Key

For more information about Ceph storing look here

CLI

See –help for all available options

See some of the examples below

Get repository PullRequest data locally

srcopsmetrics --create --is-local --repository foo_repo --entities PullRequest

which is equivalent to

srcopsmetrics -clr foo_repo -e PullRequest

Get organization PR data locally

srcopsmetrics -clo foo_org -e PullRequest

Get multiple repository PR data locally

srcopsmetrics -clr foo_repo,bar_repo -e PullRequest

Get multiple entity data locally

srcopsmetrics -clr foo_repo -e PullRequest,Issue,Commit

Meta-Information Entities Data

To know more about indicators that are extracted from data, check out Meta-Information Indicators.

Data loading using modules

>>> from srcopsmetrics.entities.pull_request import PullRequest

>>> full_repo_slug = "thoth-station/mi"
>>> pr = PullRequest(full_repo_slug)

>>> # for local data in default mi data path
>>> data = load_previous_knowledge(is_local=True)
>>> data.head()
                                       title                                               body size  ...   changed_files     first_review_at    first_approve_at
id                                                                                                        ...
97                   ⬆️ Bump rsa from 4.0 to 4.7  Bumps [rsa](https://github.com/sybrenstuvel/py...    L  ...  [Pipfile.lock]                 NaT                 NaT
96              ⬆️ Bump pyyaml from 5.3.1 to 5.4  Bumps [pyyaml](https://github.com/yaml/pyyaml)...    L  ...  [Pipfile.lock]                 NaT                 NaT
95                   ⬆️ Bump rsa from 4.0 to 4.1  Bumps [rsa](https://github.com/sybrenstuvel/py...    L  ...  [Pipfile.lock]                 NaT                 NaT
94  Automatic update of dependencies by Kebechet  Kebechet has updated the depedencies to the la...    L  ...  [Pipfile.lock] 2021-03-22 08:00:14 2021-03-22 08:00:14
93  Automatic update of dependencies by Kebechet  Kebechet has updated the depedencies to the la...    L  ...  [Pipfile.lock] 202

Any other entity is loaded in the similar way. If you intend to load remote data from Ceph, all of the Ceph variables need to be specified (see more in Setup section).

How to contribute

Always feel free to open new Issues or engage in already existing ones!

Custom Entities & Metrics

If you want to contribute by adding new entity or metric that will be analysed from GitHub repositories, feel free to open up an Issue and describe why do you think this new entity should be analysed and what are the benefits of doing so according to the goal of thoth-station/mi project.

After creating Issue, you can wait for the response of thoth-station devs Do not forget to reference the Issue in your Pull Request.

Implementation

Look at Template entity to get an idea for requirements that need to be satisfied for custom entity implementation.