Skip to main content

Command line library and Python client for Splitgraph, a version control system for data

Project description

Splitgraph

Build status Coverage Status PyPI version Discord chat room Follow

Splitgraph is a tool for building, versioning, querying and sharing datasets that works on top of PostgreSQL and integrates seamlessly with anything that uses PostgreSQL.

This repository contains most of the core code for the Splitgraph library, the sgr command line client and the Splitgraph Engine.

See https://www.splitgraph.com/docs/getting-started/introduction for the full docs.

Installation

You will need access to Docker as Splitgraph uses it to run the Splitgraph Engine.

For Linux and OSX, there's a single script:

$ bash -c "$(curl -sL https://github.com/splitgraph/splitgraph/releases/latest/download/install.sh)"

This script downloads the sgr binary and sets up the Splitgraph Engine Docker container.

Alternatively, you can get the sgr single binary from the releases page and run sgr engine add to create an engine.

See the installation guide for more installation methods.

Quick start guide

You can follow the quick start guide that will guide you through the basics of using Splitgraph with public and private data.

Alternatively, Splitgraph comes with plenty of examples to get you started.

If you're stuck or have any questions, check out the documentation or join our Discord channel!

Setting up a development environment

  • Splitgraph requires Python 3.6 or later.
  • Install Poetry: curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python to manage dependencies
  • Install pre-commit hooks (we use Black to format code)
  • git clone https://github.com/splitgraph/splitgraph.git
  • poetry install
  • To build the engine Docker image: cd engine && make

Running tests

The test suite requires docker-compose. You will also need to add these lines to your /etc/hosts or equivalent:

127.0.0.1       local_engine
127.0.0.1       remote_engine
127.0.0.1       objectstorage

To run the core test suite, do

docker-compose -f test/architecture/docker-compose.core.yml up -d
poetry run pytest -m "not mounting and not example"

To run the test suite related to "mounting" and importing data from other databases (PostgreSQL, MySQL, Mongo), do

docker-compose -f test/architecture/docker-compose.core.yml -f test/architecture/docker-compose.core.yml up -d  
poetry run pytest -m mounting

Finally, to test the example projects, do

# Example projects spin up their own engines
docker-compose -f test/architecture/docker-compose.core.yml -f test/architecture/docker-compose.core.yml down -v
poetry run pytest -m example

All of these tests run in CI.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

splitgraph-0.1.1.tar.gz (192.2 kB view hashes)

Uploaded Source

Built Distribution

splitgraph-0.1.1-py3-none-any.whl (225.3 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page