Slice DB
Overview
SliceDB is a tool for capturing and restoring a subset of a PostgreSQL database. It also supports scrubbing sensive data.
Install
Pip
pip3 install slice-db
Docker
docker pull rivethealth/slicedb
Usage
For all commands and options, see Usage.
Basic example
First, query a database to create a schema file.
slicedb schema > schema.yml
Second, dump a slice:
slicedb dump --root public.example 'WHERE id IN (7, 56, 234)' --schema schema.yml > slice.zip
Third, restore that slice into another database:
slicedb restore < slice.zip
For a complete working example, see Example.
Connection
Use the libpq environment variables to configure the connection.
PGHOST=myhost slicedb schema > slice.yml
Dump
See dump.yml for the JSONSchema.
Output formats
SliceDB can produce multiple formats:
- slice - ZIP archive. This can be restored with
slicedb restore. - sql - SQL file. This can be restored with
psqlor another client. If restoring into existing schema, foreign keys must first be disabled, e.g.SET session_replication_role = replica.
Output content
Schema can optionally be included. Restoring with schema requires an existing empty database.
Schema
The schema command uses foreign keys to infer relationships between tables. It
is a suggested starting point.
You may want to prune the slice by removing relationships, or expand the slice by adding relationships that don't have explicit foreign keys.
slicedb schema-filter can help modify the schema, or generic JSON tools like
jq.
Algorithm
The slicing process works as follows:
-
Starting with the root table, query the physical IDs (ctid) of rows.
-
Add the row IDs to the existing list.
-
For new IDs, process each of the adjacent tables, using them as the current root.
Do this in parallel, using pg_export_snapshot() to guarantee a consistent
snapshot across workers.
Performance
Hundreds of thousands of rows can be exported in only a few minutes and several dozen MBs of memory.
Transformation
See transform.yml for the JSONSchema.
Replacements are deterministic for a given pepper. By default, the pepper is
randomly generated each run. You may specify it as --pepper. Note that
possession of the pepper makes the data guessable.
Transformation may operate on an existing slice (TODO), or happen during the dump.
Configuration
Transforms are specified by:
class, the Python classconfig, transform-specific optionsmodule, defaults toslice_db.transforms
The name given to the transform is appended to the global pepper.
Custom transforms
To create custom transforms, implement slice_db.transform.Transform, expose
the class on a module, and install the module so that is accessible by
slicedb.
Built-in transforms
The slice_db.transforms package has many common transforms.
See transforms.md for the full list.
Restore
SliceDB can restore slices into existing databases. In practice, this should normally be an empty existing database.
Cycles
Foreign keys may form a cycle only if at least one foreign key in the cycle is deferrable.
That foreign key will be deferred during restore.
A restore may happen in a single transaction or not. Parallelism requires multiple transactions.
Not supported
- Multiple databases
- Databases other than PostgreSQL
Developing
Install: make install
Format: make format
Publishing
- Update slice_db/version.py
- Commit
git add slice_db/version.py && git commit -m "Version <version>" - Tag
git tag v<version> - Push
git push origin master --tags - Publish to PyPI:
make upload
4 maintainers
Metadata
4 maintainers
Release files for slice-db 5.1.18
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| slice_db-5.1.18.tar.gz | 407.1 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| slice_db-5.1.18-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 819.8 kB
Release files / slice_db-5.1.18.tar.gz
| Download URL | slice_db-5.1.18.tar.gz |
|---|---|
| Size | 407.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
e1c71dadfaddf18ff93cf609c91dcb41ddb4a9e9bf2b71a3fef8be8582226c06
|
|
BLAKE2b-256 checksum How to use checksums |
27d34b4eb0e21ce492e276b88eb1da0a613642e32559e91e24df71eb5979cbd2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.2
|
Release files / slice_db-5.1.18-py3-none-any.whl
| Download URL | slice_db-5.1.18-py3-none-any.whl |
|---|---|
| Size | 412.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
945e809f9792ad648ddb57ff70a24a5f4704d6d569fee57e2545b9c79db32c6b
|
|
BLAKE2b-256 checksum How to use checksums |
196ce8fc8f0b74c92be2862dba4d222c27ffe3f2f5ec1294c9a16548d4db0e71
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.2
|
