Copy data from any source to any destination without any code
ingestr is a command-line app that allows you to ingest data from any source into any destination using simple command-line flags, no code necessary.
- ✨ copy data from your database into any destination
- ➕ incremental loading:
append,mergeordelete+insert - 🐍 single-command installation
ingestr takes away the complexity of managing any backend or writing any code for ingesting data, simply run the command and watch the data land on its destination.
Installation
You can install ingestr using the install script:
curl -LsSf https://getbruin.com/install/ingestr | sh
Alternatively, you can install it with pip:
pip install ingestr
The pip package can also be used from Python. Install the SDK extra for Python data ingestion:
pip install 'ingestr[sdk]'
Python rows, generators, and DataFrames are sent to the ingestr CLI binary as Arrow IPC streams by default. The pip package downloads and caches the matching GitHub release binary on first use:
import ingestr
ingestr.ingest(
[{"id": 1, "name": "Ada"}, {"id": 2, "name": "Grace"}],
dest_uri="duckdb:///tmp/warehouse.duckdb",
dest_table="main.people",
)
DataFrames and yielded data use the same Arrow stream transport:
ingestr.ingest(df, dest_uri="duckdb:///tmp/warehouse.duckdb", dest_table="main.events")
def events():
yield [{"id": 1, "event": "signup"}]
yield [{"id": 2, "event": "purchase"}]
ingestr.ingest(events, dest_uri="postgresql://...", dest_table="public.events")
For push-style code, omit the data argument and use ingest as a context manager. The context value accepts the same shapes as ingestr.ingest(data, ...):
with ingestr.ingest(dest_uri="postgresql://...", dest_table="public.events") as ingest:
for response in client.list_events():
ingest(response["items"])
For very large already-materialized data, use the existing mmap Arrow IPC file transport:
ingestr.ingest(df, dest_uri="duckdb:///tmp/warehouse.duckdb", dest_table="main.events", transport="mmap")
For full CLI pass-through, use ingestr.run(["ingest", "--source-uri", "...", "--dest-uri", "...", "--source-table", "..."]), or ingestr.run_cli(...) for keyword arguments that map to CLI flags.
Quickstart
ingestr ingest \
--source-uri 'postgresql://admin:admin@localhost:8837/web?sslmode=disable' \
--source-table 'public.some_data' \
--dest-uri 'bigquery://<your-project-name>?credentials_path=/path/to/service/account.json' \
--dest-table 'ingestr.some_data'
That's it.
This command:
- gets the table
public.some_datafrom the Postgres instance. - uploads this data to your BigQuery warehouse under the schema
ingestrand tablesome_data.
Documentation
You can see the full documentation here.
Community
Join our Slack community here.
Contributing
Pull requests are welcome. However, please open an issue first to discuss what you would like to change. We maybe able to offer you help and feedback regarding any changes you would like to make.
Supported sources & destinations
| Source | Destination | CDC | |
|---|---|---|---|
| Databases | |||
| Apache Iceberg | - | ✅ | - |
| Apache Pulsar | ✅ | - | - |
| AWS Athena | ✅ | ✅ | - |
| AWS Redshift | ✅ | ✅ | - |
| Amazon SQS | ✅ | - | - |
| Azure Event Hubs | ✅ | - | - |
| Cassandra | ✅ | ✅ | - |
| ClickHouse | ✅ | ✅ | - |
| Cloudflare D1 | - | ✅ | - |
| Couchbase | ✅ | - | - |
| CrateDB | ✅ | ✅ | - |
| Databricks | ✅ | ✅ | - |
| DuckDB | ✅ | ✅ | - |
| DynamoDB | ✅ | ✅ | - |
| Elasticsearch | ✅ | ✅ | - |
| GCP Spanner | ✅ | - | - |
| Google BigQuery | ✅ | ✅ | - |
| Google Cloud Pub/Sub | ✅ | - | - |
| HTTP | ✅ | - | - |
| IBM Db2 | ✅ | - | - |
| InfluxDB | ✅ | - | - |
| Kafka | ✅ | - | - |
| Local Avro file | ✅ | - | - |
| Local CSV file | ✅ | ✅ | - |
| Local JSON file | ✅ | - | - |
| Local JSONL file | ✅ | ✅ | - |
| Local Parquet file | ✅ | ✅ | - |
| MaxCompute | ✅ | ✅ | - |
| Microsoft Fabric | ✅ | ✅ | - |
| Microsoft OneLake | - | ✅ | - |
| Microsoft SQL Server | ✅ | ✅ | ✅ |
| MongoDB | ✅ | ✅ | ✅ |
| MotherDuck | ✅ | ✅ | - |
| MQTT | ✅ | - | - |
| MySQL | ✅ | ✅ | ✅ |
| NATS | ✅ | - | - |
| Oracle | ✅ | ✅ | - |
| PlanetScale | ✅ | ✅ | ✅ |
| Postgres | ✅ | ✅ | ✅ |
| RabbitMQ | ✅ | - | - |
| Redis | ✅ | - | - |
| SAP Hana | ✅ | ✅ | - |
| Snowflake | ✅ | ✅ | - |
| Socrata | ✅ | - | - |
| SQLite | ✅ | ✅ | - |
| StarRocks | ✅ | ✅ | - |
| Synapse | - | ✅ | - |
| Trino | ✅ | ✅ | - |
| Vertica | ✅ | ✅ | - |
| Vitess | ✅ | ✅ | ✅ |
| Platforms | |||
| 2Checkout | ✅ | - | - |
| ABRA Flexi | ✅ | - | - |
| Adjust | ✅ | - | - |
| Adapty | ✅ | - | - |
| Airtable | ✅ | - | - |
| Allium | ✅ | - | - |
| Amazon Kinesis | ✅ | - | - |
| Amplitude | ✅ | - | - |
| Anthropic | ✅ | - | - |
| API-Football | ✅ | - | - |
| AppsFlyer | ✅ | - | - |
| Apple Ads | ✅ | - | - |
| Apple App Store | ✅ | - | - |
| Applovin | ✅ | - | - |
| Applovin Max | ✅ | - | - |
| Asana | ✅ | - | - |
| Attio | ✅ | - | - |
| Azure Data Lake Storage Gen2 | ✅ | ✅ | - |
| BallDontLie FIFA | ✅ | - | - |
| BambooHR | ✅ | - | - |
| Braze | ✅ | - | - |
| Bruin | ✅ | - | - |
| Chargebee | ✅ | - | - |
| Chess.com | ✅ | - | - |
| CleverTap | ✅ | ✅ | - |
| ClickUp | ✅ | - | - |
| Cloudflare Radar | ✅ | - | - |
| Cursor | ✅ | - | - |
| Customer.io | ✅ | - | - |
| Deel | ✅ | - | - |
| Docebo | ✅ | - | - |
| Dune | ✅ | - | - |
| ESPN | ✅ | - | - |
| Facebook Ads | ✅ | - | - |
| FastSpring | ✅ | - | - |
| Fireflies | ✅ | - | - |
| Fluxx | ✅ | - | - |
| football-data.org | ✅ | - | - |
| Frankfurter | ✅ | - | - |
| Freshdesk | ✅ | - | - |
| FundraiseUp | ✅ | - | - |
| G2 | ✅ | - | - |
| GitHub | ✅ | - | - |
| GitLab | ✅ | - | - |
| Google Ads | ✅ | - | - |
| Google Analytics | ✅ | - | - |
| Google Cloud Storage (GCS) | ✅ | ✅ | - |
| Google Search Console | ✅ | - | - |
| Google Sheets | ✅ | ✅ | - |
| Gorgias | ✅ | - | - |
| Granola | ✅ | - | - |
| Hostaway | ✅ | - | - |
| HubSpot | ✅ | ✅ | - |
| Indeed | ✅ | - | - |
| Intercom | ✅ | - | - |
| Internet Society Pulse | ✅ | - | - |
| Jira | ✅ | - | - |
| JobTread | ✅ | - | - |
| Kalshi | ✅ | - | - |
| Klaviyo | ✅ | - | - |
| Linear | ✅ | - | - |
| LinkedIn Ads | ✅ | - | - |
| Lumify | ✅ | - | - |
| Mailchimp | ✅ | - | - |
| Manifold | ✅ | - | - |
| Mixpanel | ✅ | - | - |
| Monday | ✅ | - | - |
| Notion | ✅ | - | - |
| Okta | ✅ | - | - |
| OpenAI | ✅ | - | - |
| Paddle | ✅ | - | - |
| Payrails | ✅ | - | - |
| Personio | ✅ | - | - |
| PhantomBuster | ✅ | - | - |
| ✅ | - | - | |
| Pipedrive | ✅ | - | - |
| Plus Vibe AI | ✅ | - | - |
| Polymarket | ✅ | - | - |
| PostHog | ✅ | - | - |
| Primer | ✅ | - | - |
| QuickBooks | ✅ | - | - |
| Recurly | ✅ | - | - |
| Reddit Ads | ✅ | - | - |
| RevenueCat | ✅ | - | - |
| RIPEstat | ✅ | - | - |
| S3 | ✅ | ✅ | - |
| Salesforce | ✅ | - | - |
| SatisMeter | ✅ | - | - |
| SendGrid | ✅ | - | - |
| SFTP | ✅ | - | - |
| SharePoint | ✅ | - | - |
| Shopify | ✅ | - | - |
| Sklik | ✅ | - | - |
| Slack | ✅ | - | - |
| Smartsheet | ✅ | - | - |
| Snapchat Ads | ✅ | - | - |
| Solidgate | ✅ | - | - |
| Square | ✅ | - | - |
| Stripe | ✅ | - | - |
| Sumble | ✅ | - | - |
| SurveyMonkey | ✅ | - | - |
| TikTok Ads | ✅ | - | - |
| Trello | ✅ | - | - |
| Trustpilot | ✅ | - | - |
| Twenty | ✅ | - | - |
| Twilio | ✅ | - | - |
| Typeform | ✅ | - | - |
| Wise | ✅ | - | - |
| Wistia | ✅ | - | - |
| Zendesk | ✅ | - | - |
| Zoom | ✅ | - | - |
Feel free to create an issue if you'd like to see support for another source or destination.
License
ingestr is source-available under the Functional Source License 1.1, with Apache 2.0 as the future license. You can use ingestr freely for internal production use, development, testing, education, research, and professional services. You cannot use ingestr to offer a competing commercial ingestion, ELT, connector, or managed data pipeline product/service.
Each version becomes Apache 2.0 two years after release.
1 maintainer
Metadata
1 maintainer
Release files for ingestr 1.1.73
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| ingestr-1.1.73-py3-none-any.whl | Python 3 | none | any | Details |
Release files / ingestr-1.1.73-py3-none-any.whl
| Download URL | ingestr-1.1.73-py3-none-any.whl |
|---|---|
| Size | 18.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
dad15bc1b0c0c1eaef746f885c89a4986c66a64941660a0e3971f106ee221db8
|
|
BLAKE2b-256 checksum How to use checksums |
03c4ec864b94260069470c94f0e6e0a84a10f7838bcc0a21e8ceb7f5196f81d7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.17
|
Security for ingestr 1.1.73
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
