Read infrastructure data from your cloud and export it to a SQL database.
Project description
cloud2sql
Read infrastructure data from your cloud and export it to a SQL database.
Installation
Python 3.9 or higher is required. It is recommended to use a separate virtual environment for this project. You can create one with the following command:
python3 -m venv venv --prompt "cloud2sql"
source venv/bin/activate
Once the virtual environment is activated, you can install cloud2sql:
pip install cloud2sql[all]
If you only require support for a specific database, instead of cloud2sql[all]
you can choose between cloud2sql[snowflake]
, cloud2sql[parquet]
, cloud2sql[postgresql]
, cloud2sql[mysql]
.
Usage
The sources and destinations for cloud2sql
are configured via a configuration file. Create your own configuration by adjusting the config template file.
You can safely delete the sections that are not relevant to you (e.g. if you do not use AWS, you can delete the aws
section).
All sections refer to cloud providers and are enabled if a configuration section is provided.
The following databases are currently supported:
SQLite
destinations:
sqlite:
database: /path/to/database.db
PostgreSQL
destinations:
postgresql:
host: 127.0.0.1
port: 5432
user: cloud2sql
password: changeme
database: cloud2sql
args:
key: value
MySQL
destinations:
mysql:
host: 127.0.0.1
port: 3306
user: cloud2sql
password: changeme
database: cloud2sql
args:
key: value
MariaDB
destinations:
mariadb:
host: 127.0.0.1
port: 3306
user: cloud2sql
password: changeme
database: cloud2sql
args:
key: value
Snowflake
destinations:
snowflake:
host: myorg-myaccount
user: cloud2sql
password: changeme
database: cloud2sql/public
args:
warehouse: compute_wh
role: accountadmin
Apache Parquet
destinations:
parquet:
path: /path/to/parquet/files
batch_size: 100_000
My database is not listed here
cloud2sql uses SQLAlchemy to connect to the database. If your database is not listed here, you can check if it is supported in SQLAlchemy Dialects. Install the relevant driver and use the connection string from the documentation.
Example
We use a minimal configuration example and export the data to a SQLite database. The example uses our AWS default credentials and the default kubernetes config.
cloud2sql --config config-example.yaml
Local Development
Create a local development environment with the following command:
make setup
source venv/bin/activate
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for cloud2sql-0.6.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | df1e81bb9caab7c567698f013852a9ceadaac679fbbafe973e254d2809f92b50 |
|
MD5 | 21cc1cd2413d036ade30b4a8e493f7be |
|
BLAKE2b-256 | 3b4200e18d4a1a05c5865f1494ba938bb83f64a0904d8552ac7cdc5ea66493c2 |