Skip to main content

ingestr is a command-line application that ingests data from various sources and stores them in any database.

Project description

Ingest & copy data from any source to any destination without any code


Ingestr is a command-line application that allows you to ingest data from any source into any destination using simple command-line flags, no code necessary.

  • ✨ copy data from your database into any destination
  • ➕ incremental loading: append, merge or delete+insert
  • 🐍 single-command installation

ingestr takes away the complexity of managing any backend or writing any code for ingesting data, simply run the command and watch the data land on its destination.

Installation

pip install ingestr

Quickstart

ingestr ingest \
    --source-uri 'postgresql://admin:admin@localhost:8837/web?sslmode=disable' \
    --source-table 'public.some_data' \
    --dest-uri 'bigquery://<your-project-name>?credentials_path=/path/to/service/account.json' \
    --dest-table 'ingestr.some_data'

That's it.

This command will:

  • get the table public.some_data from the Postgres instance.
  • upload this data to your BigQuery warehouse under the schema ingestr and table some_data.

Documentation

You can see the full documentation here.

Community

Join our Slack community here.

Supported Sources & Destinations

Source Destination
Databases
Postgres
BigQuery
Snowflake
Redshift
Databricks
DuckDB
Microsoft SQL Server
Local CSV file
MongoDB
Oracle
SAP Hana
SQLite
MySQL
Platforms
Gorgias
Google Sheets
Notion
Shopify

More to come soon!

Acknowledgements

This project would not have been possible without the amazing work done by the SQLAlchemy and dlt teams. We relied on their work to connect to various sources and destinations, and built ingestr as a simple, opinionated wrapper around their work.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

ingestr-0.7.3.tar.gz (218.1 kB view details)

Uploaded Source

Built Distribution

ingestr-0.7.3-py3-none-any.whl (56.3 kB view details)

Uploaded Python 3

File details

Details for the file ingestr-0.7.3.tar.gz.

File metadata

  • Download URL: ingestr-0.7.3.tar.gz
  • Upload date:
  • Size: 218.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.11.6

File hashes

Hashes for ingestr-0.7.3.tar.gz
Algorithm Hash digest
SHA256 93fc3e95181c89218c5a98bcc2800e50b570950a013aaf8c788021689482a792
MD5 e4de56bf4ab1c09495839de36b17db6d
BLAKE2b-256 e79544041de775373227ac1b9d0e8b350d068d310d47ae6944014f07e194aeb0

See more details on using hashes here.

File details

Details for the file ingestr-0.7.3-py3-none-any.whl.

File metadata

  • Download URL: ingestr-0.7.3-py3-none-any.whl
  • Upload date:
  • Size: 56.3 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.11.6

File hashes

Hashes for ingestr-0.7.3-py3-none-any.whl
Algorithm Hash digest
SHA256 4378cbb2314b5c7cd97190560d0a1ec475aeca83c0045cdc491ac6d84fd701d8
MD5 418fc40331a76b234ca1c79cab2add23
BLAKE2b-256 a0d1909ab9f193dd0b9b330cf3dbf9aa052ae21eaedee824cc640737f8f78d24

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page