pyrepscan

A Git Repository Leaks Scanner Python library written in C++

license Python Build

Table of Contents
About The Project
Usage
License
Contact

About The Project

PyRepScan is a python library written in C++. The library uses libgit2 for repository parsing and traversing, re2 for regex pattern matching and taskflow for concurrency. The library was written to achieve high performance and python bindings.

Built With

Performance

CPU

Library	Time	Improvement Factor
PyRepScan	2.18s	1.0x
gitleaks	63.0s	28.9x

Prerequisites

In order to compile this package you should have GCC & Python development package installed.

Fedora

sudo dnf install python3-devel gcc-c++ libgit2-devel re2-devel

Ubuntu 18.04

sudo apt install python3-dev g++-9 libgit2-dev libre2-dev

Installation

pip3 install PyRepScan

Usage

import pyrepscan

grs = pyrepscan.GitRepositoryScanner()

# Adds a specific rule, can be called multiple times or none
grs.add_rule(
    name='First Rule',
    match_pattern=r'''(-----BEGIN PRIVATE KEY-----)''',
    match_whitelist_patterns=[],
    match_blacklist_patterns=[],
)
# Compiles the rules. Should be called only once after all the rules were added
grs.compile_rules()

# Add file extensions to ignore during the search
grs.add_ignored_file_extension(
    file_extension='bin',
)
grs.add_ignored_file_extension(
    file_extension='jpg',
)

# Add file paths to ignore during the search. Free text is allowed
grs.add_ignored_file_path(
    file_path='site-packages',
)
grs.add_ignored_file_path(
    file_path='node_modules',
)

# Scans a repository
results = grs.scan(
    repository_path='/repository/path',
    branch_glob_pattern='*',
)

# Results is a list of dicts. Each dict is in the following format:
{
    'rule_name': 'First Rule',
    'author_email': 'author@email.email',
    'author_name': 'Author Name',
    'commit_id': '1111111111111111111111111111111111111111',
    'commit_message': 'The commit message',
    'commit_time': '2020-01-01T00:00:00e',
    'file_path': 'full/file/path',
    'file_oid': '47d2739ba2c34690248c8f91b84bb54e8936899a',
    'match': 'The matched group',
}

# Fetch the file_oid full content
file_content = grs.get_file_content(
    repository_path='/repository/path',
    file_oid='47d2739ba2c34690248c8f91b84bb54e8936899a',
)

# file_content
b'binary data'

# Creating a RulesManager directly
rules_manager = pyrepscan.RulesManager()

# For testing purposes, check your regexes pattern using check_pattern function
rules_manager.check_pattern(
    content='some content1 to check, another content2 in the same line\nanother content3 in another line\n',
    pattern=r'(content\d)',
)

# Results are the list of captured matches
[
    'content1',
    'content2',
    'content3',
]

License

Distributed under the MIT License. See LICENSE for more information.

Contact

Gal Ben David - gal@intsights.com

Project Link: https://github.com/intsights/PyRepScan

Metadata

Release files for pyrepscan 0.3.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pyrepscan 0.3.1
File	Size	Uploaded
PyRepScan-0.3.1.tar.gz	267.5 kB	Jul 6, 2020	Details

Release files / PyRepScan-0.3.1.tar.gz

Download URL	PyRepScan-0.3.1.tar.gz
Size	267.5 kB
Tags	Source
SHA-256 checksum How to use checksums	`a127f39203c1becef5cd51082fd46aef782b0064d21a2fce5cd3b5ff94f5beed`
BLAKE2b-256 checksum How to use checksums	`c64ce7f3682ff41fae2ebe15e54a857d5b87d1033460fb7c5304d8030a3342d7`
Upload date	Jul 6, 2020
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/47.1.1 requests-toolbelt/0.9.1 tqdm/4.47.0 CPython/3.8.3`