Skip to main content

Datacycle

Getting started

cp .env.example .env
vim .env
source .env

poetry install --extras all
poetry run datacycle
docker build -f Dockerfile -t datacycle .
docker run -it --rm --env-file .env datacycle

Mac requirements

brew install mongodb/brew/mongodb-database-tools
brew install libpq
brew link --force libpq
npm install elasticdump -g

Linux requirements

apt install -y mongo-tools
apt install -y postgresql-client
npm install elasticdump -g

How to

datacycle --help
datacycle doctor

datacycle mongo "mongodb://user:password@localhost:27017/test1?authSource=admin" "mongodb://user:password@localhost:27017/test2?authSource=admin" --transform "
    transforms {
        test1 {
            before-transform {}
        }
    }
"

datacycle mongo mongodb://user:password@localhost:27017/test1?authSource=admin gs://datacycle-test/test1/snapshot --transform ops.hocon

datacycle mongo mongodb://user:password@localhost:27017/test1?authSource=admin mongodb://user:password@localhost:27017/test2?authSource=admin
datacycle mongo mongodb://user:password@localhost:27017/test1?authSource=admin gs://datacycle-test/test1/snapshot
datacycle mongo mongodb://user:password@localhost:27017/test1?authSource=admin test1/snapshot

datacycle mongo gs://datacycle-test/test1/snapshot mongodb://user:password@localhost:27017/test2?authSource=admin
datacycle mongo gs://datacycle-test/test1/snapshot gs://datacycle-test/test2/snapshot
datacycle mongo gs://datacycle-test/test1/snapshot test2/snapshot

datacycle mongo test1/snapshot mongodb://user:password@localhost:27017/test2?authSource=admin
datacycle mongo test1/snapshot gs://datacycle-test/test2/snapshot
datacycle mongo test1/snapshot test2/snapshot

Providers

Postgres

https://www.postgresql.org/docs/9.1/backup.html

  • SQL dump
  • file system snapshot
  • continuous archiving
pg_dump --clean "postgres://user:password@localhost:5432/test" | gzip > dump.gz
gunzip -c dump.gz | psql "postgres://user:password@localhost:5432/test"

Mongo

https://docs.mongodb.com/manual/core/backups/

  • BSON dump
  • file system snapshot
  • CDC
mongodump --uri="mongodb://user:password@localhost:27017/test?authSource=admin" --out=dump --numParallelCollections=10 -v --gzip
mongorestore --uri="mongodb://user:password@localhost:27017/test?authSource=admin" dump/test --numParallelCollections=10 -v --gzip

Elasticsearch

https://github.com/elasticsearch-dump/elasticsearch-dump

  • dump
elasticdump --input=https://localhost:9200 --output=$ --limit 2000 | gzip > dump.gz

Metadata

Release files for datacycle 0.0.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for datacycle 0.0.3
File Size Uploaded Provenance Actions
datacycle-0.0.3.tar.gz 9.5 kB Without provenance Details for datacycle-0.0.3.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for datacycle 0.0.3
File Interpreter ABI Platform Actions
datacycle-0.0.3-py3-none-any.whl Python 3 none any Details for datacycle-0.0.3-py3-none-any.whl

Total release size: 20.6 kB

Release files / datacycle-0.0.3.tar.gz

Download URL datacycle-0.0.3.tar.gz
Size 9.5 kB
Tags Source
SHA-256 checksum
How to use checksums
3949711c60d67c5b714c1f0ee74cccbf600528eba3490de71ad65d60357abcef
BLAKE2b-256 checksum
How to use checksums
bc83ae63e573ece463881beea36e4a583a960ead3d4b4da76c9b72b58dc6c5bb
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via poetry/1.1.12 CPython/3.10.4 Linux/5.13.0-1021-azure

Release files / datacycle-0.0.3-py3-none-any.whl

Download URL datacycle-0.0.3-py3-none-any.whl
Size 11.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
d157c7c266fd70daa9cd14a40741b46bb0d23a01f09f85cdb08cc68728283655
BLAKE2b-256 checksum
How to use checksums
0d3061ba5b470fdebdd04f44a3b164c8b153f1903344dacac51909b64ddeb51c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via poetry/1.1.12 CPython/3.10.4 Linux/5.13.0-1021-azure

Security for datacycle 0.0.3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.0.3 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page