splitgraph 0.1.1

Command line library and Python client for Splitgraph, a version control system for data

Splitgraph

Splitgraph is a tool for building, versioning, querying and sharing datasets that works on top of PostgreSQL and integrates seamlessly with anything that uses PostgreSQL.

This repository contains most of the core code for the Splitgraph library, the sgr command line client and the Splitgraph Engine.

See https://www.splitgraph.com/docs/getting-started/introduction for the full docs.

Installation

You will need access to Docker as Splitgraph uses it to run the Splitgraph Engine.

For Linux and OSX, there's a single script:

$ bash -c "$(curl -sL https://github.com/splitgraph/splitgraph/releases/latest/download/install.sh)"

This script downloads the sgr binary and sets up the Splitgraph Engine Docker container.

Alternatively, you can get the sgr single binary from the releases page and run sgr engine add to create an engine.

See the installation guide for more installation methods.

Quick start guide

You can follow the quick start guide that will guide you through the basics of using Splitgraph with public and private data.

Alternatively, Splitgraph comes with plenty of examples to get you started.

If you're stuck or have any questions, check out the documentation or join our Discord channel!

Setting up a development environment

• Splitgraph requires Python 3.6 or later.
• Install Poetry: curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python to manage dependencies
• Install pre-commit hooks (we use Black to format code)
• git clone https://github.com/splitgraph/splitgraph.git
• poetry install
• To build the engine Docker image: cd engine && make

Running tests

The test suite requires docker-compose. You will also need to add these lines to your /etc/hosts or equivalent:

127.0.0.1       local_engine
127.0.0.1       remote_engine
127.0.0.1       objectstorage

To run the core test suite, do

docker-compose -f test/architecture/docker-compose.core.yml up -d
poetry run pytest -m "not mounting and not example"

To run the test suite related to "mounting" and importing data from other databases (PostgreSQL, MySQL, Mongo), do

docker-compose -f test/architecture/docker-compose.core.yml -f test/architecture/docker-compose.core.yml up -d  
poetry run pytest -m mounting

Finally, to test the example projects, do

# Example projects spin up their own engines
docker-compose -f test/architecture/docker-compose.core.yml -f test/architecture/docker-compose.core.yml down -v
poetry run pytest -m example

All of these tests run in CI.