Skip to main content

Credential Digger

Project description

Logo

Credential Digger

Credential Digger is a Github scanning tool that identifies hardcoded credentials (Passwords, API Keys, Secret Keys, Tokens, personal information, etc). Credential Digger has a clear advantage compared to the other Github scanners in terms of False Positive reduction in the scan reports. Credential Digger is using two Machine Learning Models to identify false positivies, especially in Password identification:

  • Path Model: Identify the portion of code that contains fake credentials used for testing and example purposes (ex. Unit tests).
  • Snippet Model: Identify the portion of code used to authenticate with passwords, and distinguish between real and fake passwords.

Architecture

Credential Digger finds credentials hardcoded in a repository. The tool is composed of:

  • Postgres database
  • Python client
  • User interface

Database

The database is structured in the following way (arrows point to foreign keys).

DB Structure

Project structure

The project includes 3 components: a db (sql folder), a client (credentialdigger folder), and a user interface (ui folder).

sql

create_table.sql defines the db schema.

Note that, given the file_name and commit_hash of a discovery, both the commit and the file can be accessible at addresses:

REPO_URL/commit/COMMIT_HASH
REPO_URL/blob/COMMIT_HASH/file_name
credentialdigger

This client can be used to easily interact with the db. It offers a scanner for git repositories, based on Hyperscan (others can be implemented).

Please note that the database must be up and running.

ui

The user interface can be used to easily perform scans and flag the discoveries.

Install

  1. Prepare the .env file and edit it with the correct data

    cp .env.sample .env
    vim .env  # Insert real credentials
    
  2. Run the db and the ui using docker-compose:

    sudo docker-compose up --build
    

    Consider not to expose the db port in production.

The ui is available at http://localhost:5000/

Build from scratch

Execute step 1. as above

  1. Run the db using docker-compose:

    sudo docker-compose up --build postgres
    
  2. Install the dependencies for the client.

    sudo apt install libhyperscan-dev libpq-dev
    
  3. Install the Python requirements from the requirements.txt file.

    pip install -r requirements.txt
    
  4. Install the client

    python setup.py install
    

Pypi

A package is available on pypi. Install the client:

pip install credentialdigger

Please note that the database must be run separately.

Run the db on a different machine

In case the db and the client are run on different machines, then clone this repository on both of them.

Then, execute the steps 1. and 2. as described in the installation section above on the machine running the db, and execute the steps described in the "build from scratch" section on the machine running the client (or install the client with pip).

In case the db and the client/ui run on separate machines, the port of the db must be exposed.

In case the UI must be run on a dedicated machine, update the .env as needed, and run:

sudo docker-compose up --build credential_digger

Use machine learning models

Machine learning models can be downloaded and used to automatically filter false positive discoveries during a scan.

After installing the client, they can be downloaded as follows.

# Export the url into an environment variable
export model_name=https://...
# Download and link the model
python -m credentialdigger download model_name

NB: Don't run the download command from the installation folder of credentialdigger in order to avoid errors in linking.

Refer to the Wiki for further information.

Integrate ML Models in the UI

ML models are not supported yet in the UI "out of the box", but they can be enabled with a little effort.

To use ML models in the docker container running the ui, set their address in the .env file. If the address is correct, the model will be installed at container building time. However, the current server is not executing any model during the scan of a repository. To enable a model, it must be listed as argument in the scan function (as would be done in the client).in the ui/server.py file.

Usage (client)

from credentialdigger.cli import Client
c = Client(dbname='MYDB', dbuser='MYUSER', dbpassword='*****',
           dbhost='localhost', dbport=5432)

Wiki

Refer to the Wiki for further information.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

credentialdigger-1.0.1.tar.gz (37.1 kB view details)

Uploaded Source

Built Distribution

credentialdigger-1.0.1-py3-none-any.whl (48.8 kB view details)

Uploaded Python 3

File details

Details for the file credentialdigger-1.0.1.tar.gz.

File metadata

  • Download URL: credentialdigger-1.0.1.tar.gz
  • Upload date:
  • Size: 37.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.24.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.7.3

File hashes

Hashes for credentialdigger-1.0.1.tar.gz
Algorithm Hash digest
SHA256 904675fde1a6c4efc1524e26569c9ffe9d6f2af128adb5a62b49db756291bc33
MD5 d6778d9a1bf9beb909afa12b2dcc5e13
BLAKE2b-256 0c7922160548785c920e227d4844a0af7140a22052240dfee12349dda12062e5

See more details on using hashes here.

File details

Details for the file credentialdigger-1.0.1-py3-none-any.whl.

File metadata

  • Download URL: credentialdigger-1.0.1-py3-none-any.whl
  • Upload date:
  • Size: 48.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.24.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.7.3

File hashes

Hashes for credentialdigger-1.0.1-py3-none-any.whl
Algorithm Hash digest
SHA256 be749576edbd563420b945f0f47597f4865ac49b474f3768bd85d5d5ad705f15
MD5 4f7280774ed8de583b4a5ca5be575f92
BLAKE2b-256 0c84c625f91b54a66a7e9b824f993f903f48d2796a5083ca2a014988b0c0c639

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page