Skip to main content

Git based helpers and Report Templates

Project description

CloudMage GitUtils Python3 Utility and Reporting Package


CloudMage


PyTests     Coverage     codecov



Table of Contents



Description

In part, the purpose of this library is to be able to either automatically detect a projects configured Git repository by searching for and parsing a .git/config file in a given file path, or to take an input consisting simply of the repositories URL (HTTP | Git formatted). The object has the repository url, provider, and any users configured within the git URL string. The Second part of this utility package is to provide a few pre-packaged reports that can be retrieved from Github such as an Open Pull Request report, showing all pull requests opened within a given repository user or organization namespace.



Road Map

Currently this library gathers data from the Githubs API, however, future development will include Gitlab and Bitbucket repository API reporting as well.



Python Version Support

This library is compatible with Python 3.6 and higher. It may work with earlier versions of Python3 but was not tested against anything earlier then 3.6. As Python 2.x is soon to be end of life, backward compatibility was not taken into consideration.



Package Installation

This library has been published to PyPi and can be installed via normal python package manager conventions such as pip or poetry.


pip3 install cloudmage-gitutils



Package Dependencies

This package installs and imports the following python modules during installation:

  • pyGithub
  • progress
  • cloudmage.jinjautils

Additionally the package takes advantage of the following built in python modules:

  • os
  • sys
  • json
  • inspect
  • datetime



GitConfigParser Class

This class takes a directory path argument, which it uses as a target directory to search for a .git/config file. If a file is found, then the class will parse the URL from the config, and determines the git platform provider from the parsed URL path. This data is then used to return back an object instance with properties set to the parsed values.


GitConfigParser Constructor Arguments


The following arguments can be used to instantiate a new object instance:


path A valid directory path where a .git/config file can be found.
Must be valid directory path, checked with os.path.exists()
required true
type str

verbose Enables verbose mode.   [true=enable   false=disable]
required false
type bool
default false (disabled)

log Redirects object standard log messaging to provided log object.
required false
type obj
default None (log to stdout, stderr if verbose=true)



GitConfigParser Attributes and Properties


The following attributes/properties are available to an instantiated object instance. Any of the attributes or properties can be accessed with standard object dot notation as in the example: verbose_mode = GitConfigParserObj.verbose


verbose Verbose setting that controls logging level within the object.   [true=enabled, false=disabled]
returns true or false (enabled or disabled)
type bool
instantiated value false

url Verbose setting that controls logging level within the object.   [true=enabled, false=disabled]
returns url string -> https://github.com/namespace/repository
type str
instantiated value URL string parsed from .git/config in directory path specified during object instantiation

provider The parsed provider (github.com, gitlab.com, or bitbucket.org) from a given URL string
returns Provider string -> github.com
type str
instantiated value Provider string parsed from object url property during object instantiation

user If a user was used in the config url, then the value of the user will be assigned to this property
returns User string -> username
type str
instantiated value User string parsed from object url property during object instantiation

log The class logger. Will either write directly to stdout, stderr, or to a lob object if passed into the object constructor during object instantiation
returns Log Event Stream
type str
instantiated value Logs written to stdout, stderr



GitConfigParser Available Methods


The following methods are available to an instantiated object instance. Some of the methods are simply setter methods that run the logic required to discover and then set one of the above instance properties.


verbose

Setter method for verbose property that enables or disables verbose mode


parameter type required arg info
verbose bool true True enables verbose logging, False disables it

Examples:

# Getter method
verbose_setting = GitConfigParserObj.verbose

# Setter method
GitConfigParserObj.verbose = True



log

Method to enable logging throughout the class. Log messages are sent to the log method providing the log message, the message type being one of [debug, info, warning, error], and finally the function or method id that is automatically derived within the function or method using the python inspect module. If a log object such as a logger or an already instantiated log object instance was passed to the class constructor during the objects instantiation, then all logs will be written to the provided log object. If no log object was provided during instantiation then all debug, info, and warning logs will be written to stdout, while any encountered error log entries will be written to stderr. Note that debug or verbose mode needs to be enabled to receive the event log stream.


arg type required arg info
log_msg str true The actual message being sent to the log method
log_type str true The type of message that is being sent to the log method, one of [debug, info, warning, error]
log_id str true A string value identifying the sender method or function, consisting of the method or function name

Examples:

def my_function():
  __function_id = inspect.stack()[0][3]
  GitConfigParserObj.log(
    f"{__function_id} called.",
    'info',
    __function_id
  )



url

Setter method for url property that will search the object instances set directory path and look for a .git/config directory in that path. If found, then the setter method will parse the .git/config file and look for a URL line to parse and extract the URL. It will then update the object property with the parsed value. If this method is called post instantiation, then a valid directory path must be provided as an argument.


parameter type required arg info
config_path str true Must be a valid directory path. This value is checked by os.path.exists() and will write an error to the log if the provided argument directory path does not exist

Examples:

GitConfigParserObj.url = /home/projects/my_project_directory



provider

Setter method for provider property that will search the object instances updated url property for a valid git provider string. Currently the provider setter will look specifically for a value that matches one of [github.com, gitlab.com, bitbucket.org]. If this method is called post instantiation, then a valid git repository url must be provided as an argument. The provider property setter will parse either an http, https, git or ssh formatted URL string. During the parse operation, if a user is identified in the provider string such as user@bitbucket.org, then the username will also be parsed and used to update the user object property.


parameter type required arg info
repository_url str true Must be a properly formatted URL string starting with one of [http, https, git, ssh] and must end in .git

Examples:

GitConfigParserObj.provider = "git@github.com:namespace/repository.git"



GitConfigParser Class Usage



GitConfigParser Default Instantiation

from cloudmage-gitutils import GitConfigParser

ProjectPath = '/Projects/MyCoolProject'
# Contains .git/config with
# url = https://github.com/GithubNamespace/MyCoolProject-Repo.git

Repo = GitConfigParser(ProjectPath)

repo_url = Repo.url
print(repo_url) # https://github.com/GithubNamespace/MyCoolProject-Repo

repo_provider = Repo.provider
print(repo_provider) # github.com

repo_user = Repo.user
print(repo_user) # None



CloudMage    Optional Verbose Class Constructor Argument:
When instantiating the class an optional verbose argument can be provided. The argument expects a bool value of either True or False. By default verbose is set to False. If verbose=True is passed during object instantiation, then debug mode is turned on allowing the class to output DEBUG, INFO, and WARNING messages to stdout, and ERROR messages to stderr.



GitConfigParser Verbose Instantiation

from cloudmage-gitutils import GitConfigParser

ProjectPath = '/Projects/MyCoolProject'
# Contains .git/config with
# url = https://github.com/GithubNamespace/MyCoolProject-Repo.git

Repo = GitConfigParser(ProjectPath, verbose=True)

repo_url = Repo.url
print(repo_url) # https://github.com/GithubNamespace/MyCoolProject-Repo

repo_provider = Repo.provider
print(repo_provider) # github.com

repo_user = Repo.user
print(repo_user) # None

# Class DEBUG, INFO, and WARNING messages will be printed to stdout, and ERROR messages will be printed to stderr



CloudMage    Optional Log Object:
When instantiating the class an optional log argument can also be provided. The argument expects an Logger object to be passed as an input. If passed then all DEBUG, INFO, WARNING, and ERROR messages will be printed to the standard log levels [log.debug(), log.info(), log.warning(), log.error()] and printed to the passed respective logger object method.



GitConfigParser Log Instantiation

from cloudmage-gitutils import GitConfigParser

# Define test log class
# This is an example log object that simply appends any DEBUG, INFO and ERROR received class messages
# to the respective log level list. Normally this would be a logger or custom log object.
class Log(object):
        """Test Log Object"""

        def __init__(self):
            """Class Constructor"""
            self.debug_logs = []
            self.info_logs = []
            self.error_logs = []

        def debug(self, message):
            """Log Debug Messages"""
            self.debug_logs.append(message)

        def info(self, message):
            """Log Debug Messages"""
            self.info_logs.append(message)

        def error(self, message):
            """Log Debug Messages"""
            self.error_logs.append(message)

# Instantiate test log class
GitLog = Log()

ProjectPath = '/Projects/MyCoolProject'
# Contains .git/config with
# url = https://github.com/GithubNamespace/MyCoolProject-Repo.git

Repo = GitConfigParser(ProjectPath, verbose=True, log=GitLog)

repo_url = Repo.url
print(repo_url) # https://github.com/GithubNamespace/MyCoolProject-Repo

repo_provider = Repo.provider
print(repo_provider) # github.com

repo_user = Repo.user
print(repo_user) # None

for items in GitLog.debug_logs:
    print(item) # Prints stored debug logs



GithubReports Class

This class will take an auth_token, and user/organization namespace name and run a query against the namespace for all open pull requests in any repositories living within that user/org namespace. Once the pull request data is returned, it can be then be used to generate the included HTML template report that will list the repository, pull request title, submitter, reviewers, creation date and days open count for each found repository.


GithubReports Constructor Arguments


The following arguments can be used to instantiate a new object instance:


verbose Enables verbose mode.   [true=enable   false=disable]
required false
type bool
default false (disabled)

auth_token A valid Github personal access token used to search issues within the user/organization namespace account.
required true
type str

<br>

log Redirects object standard log messaging to provided log object.
required false
type obj
default None (log to stdout, stderr if verbose=true)



GithubReports Attributes and Properties


The following attributes/properties are available to an instantiated object instance. Any of the attributes or properties can be accessed with standard object dot notation as in the example: verbose_mode = GithubReportsObj.verbose


verbose Verbose setting that controls logging level within the object.   [true=enabled, false=disabled]
returns true or false (enabled or disabled)
type bool
instantiated value false

auth_token Property to set/re-set the Github Authentication Token required to make calls against Github APIs.
returns string -> 0123456789109876543210
type str
instantiated value Provided Token, If token not provided during object construction, use Obj.auth_token = "<Token>"

repo_namespace The user / organization namespace that will be used as the search target
returns string -> CloudMages
type str
instantiated value None, set with Obj.repo_namespace = "<CloudMages>"

is_organization Flag to indicate if the provided namespace is an organization namespace (affects API calls).   [true=organization, false=user]
returns true or false (organization or user) namespace
type bool
instantiated value false

notify Flag to specify if, when running reports, a comment should be left on a repository that has exceeded the open days threshold.   [true=leave_comment, false=do_not_comment]
returns true or false *(leave comments or do not leave comments)
type bool
instantiated value false

open_pr_threshold Number value indicating a days threshold used for reports such as open pull requests. This value will be used to determine if a PR has exceeded desired days open.
returns int -> 5
type int
instantiated value 5 days

template_path String value of the path where reporting templates are stored.
returns str -> /Path/to/template/files
type str
instantiated value /{installation/path}/templates

log The class logger. Will either write directly to stdout, stderr, or to a lob object if passed into the object constructor during object instantiation
returns Log Event Stream
type str
instantiated value Logs written to stdout, stderr



GithubReports Available Methods


The following methods are available to an instantiated object instance. Some of the methods are simply setter methods that run the logic required to discover and then set one of the above instance properties.


verbose

Setter method for verbose property that enables or disables verbose mode


parameter type required arg info
verbose bool true True enables verbose logging, False disables it

Examples:

# Getter method
verbose_setting = GitHubReportObj.verbose

# Setter method
GitHubReportObj.verbose = True



log

Method to enable logging throughout the class. Log messages are sent to the log method providing the log message, the message type being one of [debug, info, warning, error], and finally the function or method id that is automatically derived within the function or method using the python inspect module. If a log object such as a logger or an already instantiated log object instance was passed to the class constructor during the objects instantiation, then all logs will be written to the provided log object. If no log object was provided during instantiation then all debug, info, and warning logs will be written to stdout, while any encountered error log entries will be written to stderr. Note that debug or verbose mode needs to be enabled to receive the event log stream.


arg type required arg info
log_msg str true The actual message being sent to the log method
log_type str true The type of message that is being sent to the log method, one of [debug, info, warning, error]
log_id str true A string value identifying the sender method or function, consisting of the method or function name

Examples:

def my_function():
  __function_id = inspect.stack()[0][3]
  GitConfigParserObj.log(
    f"{__function_id} called.",
    'info',
    __function_id
  )



auth_token

Setter method for auth_token property that will be used to make calls to Githubs API for authentication. The provided auth_token must be valid, and have access to read repositories and issues. An auth_token can be set during instance instantiation, using the property setter, or passed directly into a reporting method. This allows the versatility of using different tokens if necessary.


parameter type required arg info
auth_token str true Must be a valid Github auth token. The search methods will throw a BadCredentials exception if an invalid token is provided.

Examples:

# Getter method
set_auth_token = GitHubReportObj.auth_token


# Setter method
GitHubReportObj = GitHubReports(auth_token="0123456789109876543210")

# Set using property setter
GitHubReportObj.auth_token = "0123456789109876543210"

export TOKEN="0123456789109876543210"
GitHubReportObj.search_open_pulls(auth_token=$TOKEN)



repo_namespace

Setter method for repo_namespace property that will be used to specify what repository namespace will be targeted for reporting. The repository namespace must be a string value, and a valid Github user or organization namespace. The repo_namespace can be set by the property setter method, or by being passed directly into the search method as a method argument.


parameter type required arg info
repo_namespace str true Must be a valid Github user or organization namespace or reporting and/or any form of desired data collection will fail.

Examples:

# Getter method
namespace = GitHubReportObj.repo_namespace


# Setter method
GitHubReportObj = GitHubReports(auth_token="0123456789109876543210")

# Set using property setter
GitHubReportObj.repo_namespace = "CloudMages"

# Set within the reporting method call
export TOKEN="0123456789109876543210"
GitHubReportObj.search_open_pulls(
  auth_token=$TOKEN
  repo_namespace="CloudMages"
)



is_organization

Setter method for is_organization property that will be used as a flag to determine if the provided repo_namespace is a user namespace or organization namespace. This information is relevant as the APIs that will be called to fetch data for or about the specified repository as such with repository or issues searchs will be affected by this flag. If an organization namespace is passed, but not signaled that the namespace is an organization, reports or other calls to the API could fail.

Default value of this flag is set to false.


parameter type required arg info
is_organization bool false True signals organization, False signals user namepace

Examples:

# Getter method
OrgFlag = GitHubReportObj.is_organization


# Setter method
GitHubReportObj.repo_namespace = "CloudMages"
GitHubReportObj.is_organization = True
SearchIssues = GitHubReportObj.search_open_pulls()

notify

Setter method for notify property that will be used as a flag to publish a @mentioned comment on any open pull requests or issues that were found within the searched namespace if the open pull request or issue has been opened for a period of time exceeding the value set in open_pr_threshold. If an open pull request or issue is discovered, and Obj.notify is set to True, then a message that specifically @mentions the issue submitter and all listed issue reviewers will be constructed and written on any offending issue as a comment indicating that the issue has been open in access of the threshold and should be reviewed and closed immediately. If the value is set to True, then messages will be sent to the log, but will not be published or commented on the repository issue itself.

By Default this value is set to False


parameter type required arg info
notify bool false True signals notify, False signals write log only

Examples:

# Getter method
PublishComment = GitHubReportObj.notify

# Setter method
GitHubReportObj.notify = True
SearchIssues = GitHubReportObj.search_open_pulls()

open_pr_threshold

Setter method for open_pr_threshold property that is used as the threshold metric for any open pull requests or issues discovered during namespace search. If a issue or pull request is open and the created date is greater then the value that is set for this flag in days old, then the issue or pull request will be added to reporting, and can be commented on by the search method to indicate that the pull request or issue has been open for an extended amount of time and should be reviewed and / or merged / closed immediately.

By Default this value is set to 5 days


parameter type required arg info
days int false Number of days threshold for pull requests / Github Issues.

template_path

Getter only method for template_path property that is used to simply get the file location of where the modules HTML report templates are stored. This value can not, nor does not need to have its value altered as the included model templates will always be packaged in the /templates direcotry, held within the project root.

By Default this value is {/path/to/this/module}/templates/


parameter type required arg info
None N/A N/A No Parameter Value, this is a Getter method only.

Examples:

# Getter method
TemplatePath = GitHubReportObj.template_path

search_open_pulls

The search_open_pulls reporting method will search a provided namespace for all open pull requests. For each open pull request item, the pull request Name, HTML URL, Title, Body, Submitter, Reviewers, Merge Data, Creation Data, Age, and Review States will be collected and returned back as a list of dictionaries. This data can then be used with the provided module template to render into an HTML report. The report will indicate by a green background any pull requests that have been approved and are awaiting either additional approvers or the submitter. The report will also indicate with a red background in the PR Days Open field if the pull request has been open longer then the configured open_pr_threshold number of days.


parameter type required arg info
auth_token str true Must be a valid Github auth token. The search methods will throw a BadCredentials exception if an invalid token is provided.
repo_namespace str true Must be a valid Github user or organization namespace or reporting and/or any form of desired data collection will fail.

Examples:

Open_PRs = GitHubReportObj.search_open_pulls(
  auth_token=$TOKEN
  repo_namespace="CloudMages"
)



Execution Example:

Constructing search query for CloudMages repository namespace...
    Open Pull Request Threshold:          5 days
    Notifications Enabled:                False
    Repository Namespace is Organization: True
    Verbose Mode Enabled:                 True

Instantiated Github API Connector Object
Search Results: 1 open PullRequests were returned!

Open PR Search returned 1 results
Validating Search Results...

https://github.com/DaCloudMage/UnitTest-GitUtils/pull/1 Within the Open Days Limit.

Printing Collected Open Pull Request DataSet:

[{
  "id": 123456789,
  "repository": "UnitTest-GitUtils",
  "repository_url": "https://github.com/DaCloudMage/UnitTest-GitUtils",
  "number": 1,
  "submitter": "user_1",
  "reviewers": [],
  "link": "https://github.com/DaCloudMage/UnitTest-GitUtils/pull/1",
  "title": "Open pr",
  "body": "Test Open PR Report",
  "created": datetime.datetime(2020, 4, 9, 0, 29, 56),
  "age": datetime.timedelta(days=-1, seconds=66866, microseconds=735729),
  "age_days": 1,
  "state": "open",
  "is_merged": False,
  "merged": None,
  "mergable": True,
  "merge_state": "clean",
  "merged_by": None,
  "review_count": 0,
  "days_open_threshold": 5
}]

1 / 1 of the returned search results were verified as open pull requests.



GithubReports Class Usage


GithubReports Default Instantiation

from cloudmage-gitutils import GitHubReports
import os

# Fetch exported Token
AUTH_TOKEN = os.environ['GITHUB_TOKEN']

# Instantiate the object
OpenPRs = GitHubReports(auth_token=AUTH_TOKEN)
OpenPRs.repo_namespace = "CloudMages"
OpenPrs.is_organization = True
SearchOpenPRs = OpenPrs.search_open_pulls()

print(len(SearchOpenPRs))



CloudMage    Optional Verbose Class Constructor Argument:
When instantiating the class an optional verbose argument can be provided. The argument expects a bool value of either True or False. By default verbose is set to False. If verbose=True is passed during object instantiation, then debug mode is turned on allowing the class to output DEBUG, INFO, and WARNING messages to stdout, and ERROR messages to stderr.



GithubReports Verbose Instantiation

from cloudmage-gitutils import GitHubReports
import os

# Fetch exported Token
AUTH_TOKEN = os.environ['GITHUB_TOKEN']

# Instantiate the object
OpenPRs = GitHubReports(auth_token=AUTH_TOKEN, verbose=True)
OpenPRs.repo_namespace = "CloudMages"
OpenPrs.is_organization = True
SearchOpenPRs = OpenPrs.search_open_pulls()

print(len(SearchOpenPRs))

# Class DEBUG, INFO, and WARNING messages will be printed to stdout, and ERROR messages will be printed to stderr



CloudMage    Optional Log Object:
When instantiating the class an optional log argument can also be provided. The argument expects an Logger object to be passed as an input. If passed then all DEBUG, INFO, WARNING, and ERROR messages will be printed to the standard log levels [log.debug(), log.info(), log.warning(), log.error()] and printed to the passed respective logger object method.



GithubReports Log Instantiation

from cloudmage-gitutils import GitHubReports
import os

# Define test log class
# This is an example log object that simply appends any DEBUG, INFO and ERROR received class messages
# to the respective log level list. Normally this would be a logger or custom log object.
class Log(object):
        """Test Log Object"""

        def __init__(self):
            """Class Constructor"""
            self.debug_logs = []
            self.info_logs = []
            self.error_logs = []

        def debug(self, message):
            """Log Debug Messages"""
            self.debug_logs.append(message)

        def info(self, message):
            """Log Debug Messages"""
            self.info_logs.append(message)

        def error(self, message):
            """Log Debug Messages"""
            self.error_logs.append(message)

# Instantiate test log class
GitLog = Log()

# Fetch exported Token
AUTH_TOKEN = os.environ['GITHUB_TOKEN']

# Instantiate the object
OpenPRs = GitHubReports(auth_token=AUTH_TOKEN, verbose=True, log=GitLog)
OpenPRs.repo_namespace = "CloudMages"
OpenPrs.is_organization = True
SearchOpenPRs = OpenPrs.search_open_pulls()

for item in GitLog.debug_logs:
    print(item) # Prints stored debug logs



Changelog

To view the project changelog see: ChangeLog:



TheCloudMage   Contacts and Contributions

This project is owned and maintained by: @TheCloudMage


To contribute, please:

  • Fork the project
  • Create a local branch
  • Submit Changes
  • Create A Pull Request

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for cloudmage-gitutils, version 1.0.1
Filename, size File type Python version Upload date Hashes
Filename, size CloudMage_GitUtils-1.0.1-py3-none-any.whl (46.5 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size CloudMage-GitUtils-1.0.1.tar.gz (55.7 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page