Skip to main content

General toolset to backup & restore with random/filtered/anonymized data (Mongo/Postgres/GCS).

Project description


Getting started

cp .env.example .env
vim .env
source .env

poetry install --extras all
poetry run datacycle
docker build -f Dockerfile -t datacycle .
docker run -it --rm --env-file .env datacycle

Mac requirements

brew install mongodb/brew/mongodb-database-tools
brew install libpq
brew link --force libpq
npm install elasticdump -g

Linux requirements

apt install -y mongo-tools
apt install -y postgresql-client
npm install elasticdump -g

How to

datacycle --help
datacycle doctor

datacycle mongo "mongodb://user:password@localhost:27017/test1?authSource=admin" "mongodb://user:password@localhost:27017/test2?authSource=admin" --transform "
    transforms {
        test1 {
            before-transform {}

datacycle mongo mongodb://user:password@localhost:27017/test1?authSource=admin gs://datacycle-test/test1/snapshot --transform ops.hocon

datacycle mongo mongodb://user:password@localhost:27017/test1?authSource=admin mongodb://user:password@localhost:27017/test2?authSource=admin
datacycle mongo mongodb://user:password@localhost:27017/test1?authSource=admin gs://datacycle-test/test1/snapshot
datacycle mongo mongodb://user:password@localhost:27017/test1?authSource=admin test1/snapshot

datacycle mongo gs://datacycle-test/test1/snapshot mongodb://user:password@localhost:27017/test2?authSource=admin
datacycle mongo gs://datacycle-test/test1/snapshot gs://datacycle-test/test2/snapshot
datacycle mongo gs://datacycle-test/test1/snapshot test2/snapshot

datacycle mongo test1/snapshot mongodb://user:password@localhost:27017/test2?authSource=admin
datacycle mongo test1/snapshot gs://datacycle-test/test2/snapshot
datacycle mongo test1/snapshot test2/snapshot



  • SQL dump
  • file system snapshot
  • continuous archiving
pg_dump --clean "postgres://user:password@localhost:5432/test" | gzip > dump.gz
gunzip -c dump.gz | psql "postgres://user:password@localhost:5432/test"


  • BSON dump
  • file system snapshot
  • CDC
mongodump --uri="mongodb://user:password@localhost:27017/test?authSource=admin" --out=dump --numParallelCollections=10 -v --gzip
mongorestore --uri="mongodb://user:password@localhost:27017/test?authSource=admin" dump/test --numParallelCollections=10 -v --gzip


  • dump
elasticdump --input=https://localhost:9200 --output=$ --limit 2000 | gzip > dump.gz

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

datacycle-0.0.3.tar.gz (9.5 kB view hashes)

Uploaded source

Built Distribution

datacycle-0.0.3-py3-none-any.whl (11.2 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page