graphsense-lib 2.4.0

Graphsense backend lib and automation cli

graphsense-lib

A central repository for python utility functions and everything that deals with the graphsense backend. Its CLI interface can be used to control important graphsense maintainance tasks.

Use

As a library

Access the graphsense cassandra db. This requires a graphsense config (see Modules/Config)

from graphsenselib.db import DbFactory

env, currency = "test", "btc"

with DbFactory().from_config(env, currency) as db:
    print(db.transformed.get_highest_block())

Alternatively, the database can also be accessed without relying on the graphsense config file.

from graphsenselib.db import DbFactory

with DbFactory().from_name(
        raw_keyspace_name="eth_raw",
        transformed_keyspace_name="eth_transformed",
        schema_type="account",
        cassandra_nodes = ["test.cassandra1", "test.cassandra2"]
    ) as db:
    print(db.transformed.get_highest_block())

From the commandline.

Graphsenslib exposes a commandline interface - graphsence-cli.

Help can be viewed by:

> graphsense-cli --help
Usage: graphsense-cli [OPTIONS] COMMAND [ARGS]...

  Commandline interface of graphsense-lib

  graphsense-cli exposes many tools and features to manager your graphsense
  crypto-analytics database.

Options:
  -v, --verbose  One v for warning, two for info etc.
  --help         Show this message and exit.

Commands:
  config          Inspect the current configuration of graphsenselib.
  convert         Useful file convertions tools for the graphsense...
  db              Query related functions.
  delta-update    Updating the transformed keyspace from the raw keyspace.
  exchange-rates  Fetching and ingesting exchange rates.
  ingest          Ingesting raw cryptocurrency data from nodes into the...
  monitoring      Tools to monitor the graphsense infrastructure.
  schema          Creating and validating the db schema.
  version         Display the current version.
  watch           Commands for permanently watching cryptocurrency events.

  GraphSense - https://graphsense.github.io/

Modules

DB

Assess the current database state.

> graphsense-cli db --help
Usage: graphsense-cli db [OPTIONS] COMMAND [ARGS]...

  DB-management related functions.

Options:
  --help  Show this message and exit.

Commands:
  block  Special db query functions regarding blocks.
  logs   Special db query functions regarding logs.
  state  Summary Prints the current state of the graphsense database.

Schema

Extracting and validating the current database schema.

> graphsense-cli schema --help
Usage: graphsense-cli schema [OPTIONS] COMMAND [ARGS]...

  Creating and validating the db schema.

Options:
  --help  Show this message and exit.

Commands:
  create               Creates the necessary graphsense tables in Cassandra.
  show-by-currency     Prints the current db schema expected from
                       graphsenselib
  show-by-schema-type  Prints the current db schema expected from
                       graphsenselib
  validate             Validates if the expected schema matches the database.

Ingest

Loading raw data from cryptocurrency nodes into the graphsense raw keyspace.

> graphsense-cli ingest --help
Usage: graphsense-cli ingest [OPTIONS] COMMAND [ARGS]...

  Ingesting raw cryptocurrency data from nodes into the graphsense database

Options:
  --help  Show this message and exit.

Commands:
  from-node  Ingests cryptocurrency data form the client/node to the...

Delta Update

Updates the data in the transformed keyspace based on the raw keyspace

> graphsense-cli delta-update --help
Usage: graphsense-cli delta-update [OPTIONS] COMMAND [ARGS]...

  Updating the transformed keyspace from the raw keyspace.

Options:
  --help  Show this message and exit.

Commands:
  patch-exchange-rates  Rewrites the transformed exchange rate at a...
  status                Shows the status of the delta updater.
  update                Updates transformed from raw, if possible.
  validate              Validates the current delta update status and its...

Exchange Rates

Fetches exchange_rates from different sources and adds them to the database.

> graphsense-cli exchange-rates --help
Usage: graphsense-cli exchange-rates [OPTIONS] COMMAND [ARGS]...

  Fetching and ingesting exchange rates.

Options:
  --help  Show this message and exit.

Commands:
  coindesk       From coindesk.
  coinmarketcap  From coinmarketcap.

Config

Shows the currently used configuration. Configurations reside per default in ~/graphsense.yaml. It contains the cassandra and keyspace configurations for different environments. A template config can be generated via graphsense-cli config template

> graphsense-cli config --help
Usage: graphsense-cli config [OPTIONS] COMMAND [ARGS]...

  Inspect the current configuration of graphsenselib.

Options:
  --help  Show this message and exit.

Commands:
  path      Prints the path where the config is loaded from.
  show      Prints the configuration used in the environment.
  template  Generates a configuration template.

Monitoring

Helpful functions to keep an eye on the state and health of your graphsense database state.

> graphsense-cli monitoring --help
Usage: graphsense-cli monitoring [OPTIONS] COMMAND [ARGS]...

  Tools to monitor the graphsense infrastructure.

Options:
  --help  Show this message and exit.

Commands:
  get-summary  Receives a summary record of the current database state.
  notify       Sends a message to the configured handlers (e.g. a slack
               channel) by topic.

Watch (Alpha)

Provides functions to generate notifications on cryptocurrency events like value flows on certain addresses.

> graphsense-cli watch --help
Usage: graphsense-cli watch [OPTIONS] COMMAND [ARGS]...

  Commands for permanently watching cryptocurrency events.

Options:
  --help  Show this message and exit.

Commands:
  money-flows  Watches for movements money flows and generates...

Install

To install the package from the python package index run.

  > pip install graphsense-lib

If you want to install it from source run

  > make install

or

  > pip install .

Development

Caution: python 3.11 is currently not supported. Please use python3.10.

it is advised to use a virtual environment (venv) for development. Run the following command to initialize one

    > python3 -m venv .venv

and activate it (in bash) using

    > source .venv/bin/activate

For more information refer to (venv). Run

    > make dev

to initialize the dev environment. If you want to install graphsenselib in development mode run

    > make install-dev

Before committing anything to the repository please format, lint and test your code in that order. Fix all linter warnings and make sure all test are passing before a commit.

Use the following commands for that:

    > make format
    > make lint
    > make test

or equivalently run

    > make pre-commit

Some slow tests are excluded when running make test. Occasionally, one should run

    > make test-all

to run the entire test-suite.

Linting and formatting should be automatically executed on every git commit, using pre-commit.

To create the documentation please run:

    > make docs

Creating the docs need python dev dependencies to build see (Stackoverflow)

Tagging a release

To tag a new release please update the changelog first. Afterwards, update the Version numbers RELEASESEM and RELEASE in the main Makefile.

To apply the tags run

    > make tag-version

Open SSL - Errors running test suites

Some components used by graphsense-lib use OpenSSL, to provide certain hash functions. Since OpenSSL 3.0 some hash functions needed are not available anymore by default (e.g. ripemd160). This leads to errors while running the test suite. To avoid these errors, enable legacy providers in the OpenSSL config. An example of the necessary changes can be found in the "fix openssl legacy mode" step in .github/workflows/run_tests.yaml