sirmordred 0.2.12

Drive GrimoireLab tools to produce a dashboard

SirMordred |Build Status|\ |Coverage Status|
============================================

SirMordred is the tool used to coordinate the execution of the
GrimoireLab platform, via a configuration file. Below you can find
details about the different sections composing the configuration file.

General Sections
----------------

[es\_collection]
~~~~~~~~~~~~~~~~

- **arthur** (bool: False): Use arthur for collecting items from
perceval
- **arthur\_url** (str: None): URL for the arthur service
- **password** (str: None): Password for connection to Elasticsearch
- **redis\_url** (str: None): URL for the redis service
- **url** (str: http://172.17.0.1:9200): Elasticsearch URL
(**Required**)
- **user** (str: None): User for connection to Elasticsearch ###
[es\_enrichment]

- **autorefresh** (bool: True): Execute the autorefresh of identities
- **autorefresh\_interval** (int: 2): Time interval (days) to
autorefresh identities
- **password** (str: None): Password for connection to Elasticsearch
- **url** (str: http://172.17.0.1:9200): Elasticsearch URL
(**Required**)
- **user** (str: None): User for connection to Elasticsearch ###
[general]

- **bulk\_size** (int: 1000): Number of items to write in Elasticsearch
using bulk operations
- **debug** (bool: True): Debug mode (logging mainly) (**Required**)
- **log\_backup\_count** (int: 5): Number of rotate logs files to
preserve
- **log\_handler** (str: file): use rotate for rotating the logs
automatically
- **log\_max\_bytes** (int: 104857600): Max number of bytes per log
file
- **logs\_dir** (str: logs): Directory with the logs of sirmordred
(**Required**)
- **min\_update\_delay** (int: 60): Short delay between tasks (collect,
enrich ...)
- **scroll\_size** (int: 100): Number of items to read from
Elasticsearch when scrolling
- **short\_name** (str: Short name): Short name of the project
(**Required**)
- **update** (bool: False): Execute the tasks in loop (**Required**)
- **aliases\_file** (str: ./aliases.json): JSON file to define aliases
for raw and enriched indexes
- **menu\_file** (str: ./menu.yaml): YAML file to define the menus to
be shown in Kibiter
- **global\_data\_sources** (list: bugzilla, bugzillarest, confluence,
discourse, gerrit, jenkins, jira): List of data sources collected
globally, they are declared in the section 'unknown' of the
projects.json
- **retention\_hours** (int: None): the maximum number of hours wrt the
current date to retain the data ### [panels]

- **community** (bool: True): Include community section in dashboard
- **kibiter\_default\_index** (str: git): Default index pattern for
Kibiter
- **kibiter\_time\_from** (str: now-90d): Default time interval for
Kibiter
- **kibiter\_url** (str): Kibiter URL (**Required**)
- **kibiter\_version** (str: None): Kibiter version
- **kafka** (bool: False): Include KIP section in dashboard
- **github-repos** (bool: False): Enable GitHub repo stats menu
- **gitlab-issues** (bool: False): Enable GitLab issues menu
- **gitlab-merges** (bool: False): Enable GitLab merge requests menu
- **mattermost** (bool: False): Enable Mattermost menu
- **strict** (bool: True): Enable strict panels loading ### [phases]

- **collection** (bool: True): Activate collection of items
(**Required**)
- **enrichment** (bool: True): Activate enrichment of items
(**Required**)
- **identities** (bool: True): Do the identities tasks (**Required**)
- **panels** (bool: True): Load panels, create alias and other tasks
related (**Required**)
- **report** (bool: False): Generate the PDF report for a project
(alpha)
- **track\_items** (bool: False): Track specific items from a gerrit
repository ### [projects]

- **load\_eclipse** (bool: False): Load the projects from Eclipse
- **projects\_file** (str: projects.json): Projects file path with
repositories to be collected group by projects
- **projects\_url** (str: None): Projects file URL ### [report]

- **config\_file** (str: report.cfg): Config file for the report
(**Required**)
- **data\_dir** (str: report\_data): Directory in which to store the
report data (**Required**)
- **end\_date** (str: 2100-01-01): End date for the report
(**Required**)
- **filters** (list: []): General filters to be applied to all queries
- **interval** (str: quarter): Interval for the report (**Required**)
- **offset** (str: None): Date offset to be applied to start and end
- **start\_date** (str: 1970-01-01): Start date for the report
(**Required**) ### [sortinghat]

- **affiliate** (bool: True): Affiliate identities to organizations
(**Required**)
- **autogender** (bool: False): Add gender to the profiles (executes
autogender)
- **autoprofile** (list: ['customer', 'git', 'github']): Order in which
to get the identities information for filling the profile
(**Required**)
- **bots\_names** (list: []): Name of the identities to be marked as
bots
- **database** (str: sortinghat\_db): Name of the Sortinghat database
(**Required**)
- **host** (str: mariadb): Host with the Sortinghat database
(**Required**)
- **identities\_api\_token** (str: None): API token for remote
operation with GitHub and Gitlab
- **identities\_export\_url** (str: None): URL in which to export the
identities in Sortinghat
- **identities\_file** (list: []): File path with the identities to be
loaded in Sortinghat
- **identities\_format** (str: sortinghat): Format of the identities
data to be loaded
- **load\_orgs** (bool: False):
- **matching** (list: ['email']): Algorithm for matching identities in
Sortinghat (**Required**)
- **no\_bots\_names** (list: []): Name of the identities to be unmarked
as bots
- **orgs\_file** (str: None): File path with the organizations to be
loaded in Sortinghat
- **password** (str: ): Password to access the Sortinghat database
(**Required**)
- **reset\_on\_load** (bool: False): Unmerge and remove affiliations
for all identities on load
- **sleep\_for** (int: 3600): Delay between task identities executions
(**Required**)
- **strict\_mapping** (bool: True): rigorous check of values in
identities matching (i.e, well formed email addresses)
- **unaffiliated\_group** (str: Unknown): Name of the organization for
unaffiliated identities (**Required**)
- **user** (str: root): User to access the Sortinghat database
(**Required**) ### [track\_items]

- **project** (str: TrackProject): Gerrit project to track
(**Required**)
- **raw\_index\_gerrit** (str: ): Name of the gerrit raw index
(**Required**)
- **raw\_index\_git** (str: ): Name of the git raw index (**Required**)
- **upstream\_raw\_es\_url** (str: ): URL with the file with the gerrit
reviews to track (**Required**) ## Backend Sections

In this section, a template of a backend section is shown. Further
information about Perceval backends parameters are available at:

- Params details:
http://perceval.readthedocs.io/en/latest/perceval.backends.core.html
- Examples:
https://github.com/chaoss/grimoirelab-sirmordred/blob/master/tests/test\_studies.cfg

[backend-name:tag] # :tag is optional
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- **collect** (bool: True): enable/disable collection phase
- **raw\_index** (str: None): Index name in which to store the raw
items (**Required**)
- **enriched\_index** (str: None): Index name in which to store the
enriched items (**Required**)
- **studies** (list: []): List of studies to be executed
- **backend-param-1**: ..
- **backend-param-2**: ..
- **backend-param-n**: ..

Enrichment params
^^^^^^^^^^^^^^^^^

Some backend sections allow to specify specific enrichment options,
listed below.

[jenkins]
'''''''''

- **node\_regex**: regular expression for extracting node name from
``builtOn`` field. This regular expression **must contain at least
one group**. First group will be used to extract node name. More
groups are allowed but not used to extract anything else.

Studies Sections
----------------

In this section, a template of a study section is shown. A complete list
of studies parameters is available at:

- https://github.com/chaoss/grimoirelab-sirmordred/blob/master/tests/test\_studies.cfg

[studies-name:tag] # :tag is optional
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- **study-param-1**: ..
- **study-param-2**: ..
- **study-param-n**: ..

Micro Mordred
-------------

Micro Mordred is a simplified version of Mordred which omits the use of
its scheduler. Thus, Micro Mordred allows to run single Mordred tasks
(e.g., raw collection, enrichment) per execution.

Micro Mordred is located in the
`/utils <https://github.com/chaoss/grimoirelab-sirmordred/tree/master/utils/micro.py>`__
folder of this same repository. It can be executed via command line, its
parameters are summarized below:

::

--debug: execute Micro Mordred in debug mode

--raw: activate raw task
--arthur: use Arthur to collect the raw data

--enrich: activate enrich task

--identities: activate merge identities task

--panels: activate panels task

--cfg: path of the onfiguration file
--backends: list of cfg sections where the active tasks will be executed

Examples of possible executions are shown below:

::

cd .../grimoirelab-sirmordred/utils/
micro.py --raw --enrich ./setup.cfg --backends git # execute the Raw and Enrich tasks for the Git cfg section
micro.py --panels # execute the Panels task to load the Sigils panels to Kibiter

Getting started
---------------

There are 3 options to get started with SirMordred:

Source code
^^^^^^^^^^^

You will need to install ElasticSearch (6.1.0), Kibana (6.1.4) and a
MySQL/MariaDB database, and the following components:

- `SirModred <https://github.com/chaoss/grimoirelab-sirmordred>`__
- `ELK <https://github.com/chaoss/grimoirelab-elk>`__
- `King Arthur <https://github.com/chaoss/grimoirelab-kingarthur>`__
- `Perceval <https://github.com/chaoss/grimoirelab-perceval>`__
- `Perceval for
Mozilla <https://github.com/chaoss/grimoirelab-perceval-mozilla>`__
- `Perceval for
OPNFV <https://github.com/chaoss/grimoirelab-perceval-opnfv>`__
- `Perceval for
Puppet <https://github.com/chaoss/grimoirelab-perceval-puppet>`__
- `Perceval for
FINOS <https://github.com/Bitergia/grimoirelab-perceval-finos>`__
- `SortingHat <https://github.com/chaoss/grimoirelab-sortinghat>`__
- `Sigils <https://github.com/chaoss/grimoirelab-sigils>`__
- `Kidash <https://github.com/chaoss/grimoirelab-kidash>`__
- `Toolkit <https://github.com/chaoss/grimoirelab-toolkit>`__
- `Cereslib <https://github.com/chaoss/grimoirelab-cereslib>`__
- `Manuscripts <https://github.com/chaoss/grimoirelab-manuscripts>`__

Source code and docker
~~~~~~~~~~~~~~~~~~~~~~

You will have to install the GrimoireLab components listed above, and
use the following docker-compose to have ElasticSearch, Kibana and
MariaDB. Note that you can omit the ``mariadb`` section in case you have
MySQL/MariaDB already installed in your system.

.. code:: buildoutcfg

elasticsearch:
restart: on-failure:5
image: bitergia/elasticsearch:6.1.0-secured
command: elasticsearch -Enetwork.bind_host=0.0.0.0 -Ehttp.max_content_length=2000mb
environment:
- ES_JAVA_OPTS=-Xms2g -Xmx2g
ports:
- 9200:9200

kibiter:
restart: on-failure:5
image: bitergia/kibiter:secured-v6.1.4-2
environment:
- PROJECT_NAME=Development
- NODE_OPTIONS=--max-old-space-size=1000
- ELASTICSEARCH_URL=https://elasticsearch:9200
links:
- elasticsearch
ports:
- 5601:5601

mariadb:
restart: on-failure:5
image: mariadb:10.0
expose:
- "3306"
ports:
- "3306:3306"
environment:
- MYSQL_ROOT_PASSWORD=
- MYSQL_ALLOW_EMPTY_PASSWORD=yes
- MYSQL_DATABASE=test_sh
command: --wait_timeout=2592000 --interactive_timeout=2592000 --max_connections=300
log_driver: "json-file"
log_opt:
max-size: "100m"
max-file: "3"

Only docker
^^^^^^^^^^^

Follow the instruction in the GrimoireLab tutorial to have `SirMordred
in a
container <https://chaoss.github.io/grimoirelab-tutorial/sirmordred/container.html>`__

.. |Build Status| image:: https://travis-ci.org/chaoss/grimoirelab-sirmordred.svg?branch=master
:target: https://travis-ci.org/chaoss/grimoirelab-sirmordred
.. |Coverage Status| image:: https://coveralls.io/repos/github/chaoss/grimoirelab-sirmordred/badge.svg?branch=master
:target: https://coveralls.io/github/chaoss/grimoirelab-sirmordred?branch=master