Read infrastructure data from your cloud and export it to a SQL database.
Project description
Cloud2SQL 🤩
Read infrastructure data from your cloud ☁️ and export it to a SQL database 📋.
Installation
Python 3.9 or higher is required.
You can install Cloud2SQL using:
pip install --user cloud2sql[all]
If you only require support for a specific database, instead of cloud2sql[all]
you can choose between cloud2sql[snowflake]
, cloud2sql[parquet]
, cloud2sql[postgresql]
, cloud2sql[mysql]
.
Usage
The sources and destinations for cloud2sql
are configured via a configuration file. Create your own configuration by adjusting the config template file.
You can safely delete the sections that are not relevant to you (e.g. if you do not use AWS, you can delete the aws
section).
All sections refer to cloud providers and are enabled if a configuration section is provided.
In the next section you will create a YAML configuration file. Once you have created your configuration file, you can run cloud2sql
with the following command:
cloud2sql --config myconfig.yaml
Configuration
Cloud2SQL uses a YAML configuration file to define the sources
and destinations
.
Sources
AWS
sources:
aws:
# AWS Access Key ID (null to load from env - recommended)
access_key_id: null
# AWS Secret Access Key (null to load from env - recommended)
secret_access_key: null
# IAM role name to assume
role: null
# List of AWS profiles to collect
profiles: null
# List of AWS Regions to collect (null for all)
region: null
# Scrape the entire AWS organization
scrape_org: false
# Assume given role in current account
assume_current: false
# Do not scrape current account
do_not_scrape_current: false
Google Cloud
sources:
gcp:
# GCP service account file(s)
service_account: []
# GCP project(s)
project: []
Kubernetes
sources:
k8s:
# Configure access via kubeconfig files.
# Structure:
# - path: "/path/to/kubeconfig"
# all_contexts: false
# contexts: ["context1", "context2"]
config_files: []
# Alternative: configure access to k8s clusters directly in the config.
# Structure:
# - name: 'k8s-cluster-name'
# certificate_authority_data: 'CERT'
# server: 'https://k8s-cluster-server.example.com'
# token: 'TOKEN'
configs: []
DigitalOcean
sources:
digitalocean:
# DigitalOcean API tokens for the teams to be collected
api_tokens: []
# DigitalOcean Spaces access keys for the teams to be collected, separated by colons
spaces_access_keys: []
Destinations
SQLite
destinations:
sqlite:
database: /path/to/database.db
PostgreSQL
destinations:
postgresql:
host: 127.0.0.1
port: 5432
user: cloud2sql
password: changeme
database: cloud2sql
args:
key: value
MySQL
destinations:
mysql:
host: 127.0.0.1
port: 3306
user: cloud2sql
password: changeme
database: cloud2sql
args:
key: value
MariaDB
destinations:
mariadb:
host: 127.0.0.1
port: 3306
user: cloud2sql
password: changeme
database: cloud2sql
args:
key: value
Snowflake
destinations:
snowflake:
host: myorg-myaccount
user: cloud2sql
password: changeme
database: cloud2sql/public
args:
warehouse: compute_wh
role: accountadmin
Apache Parquet
destinations:
file:
path: /where/to/write/parquet/files/
format: parquet
batch_size: 100_000
CSV
destinations:
file:
path: /where/to/write/to/csv/files/
format: csv
batch_size: 100_000
My database is not listed here
Cloud2SQL uses SQLAlchemy to connect to the database. If your database is not listed here, you can check if it is supported in SQLAlchemy Dialects. Install the relevant driver and use the connection string from the documentation.
Example
We use a minimal configuration example and export the data to a SQLite database. The example uses our AWS default credentials and the default kubernetes config.
cloud2sql --config config-example.yaml
For a more in-depth example, check out our blog post.
Local Development
Create a local development environment with the following command:
make setup
source venv/bin/activate
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for cloud2sql-0.6.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bdd5cf94c9515605839834fe03e00eb5c102d0c0ffe11b6956b56d953e3767da |
|
MD5 | 931f89a9c03827de55f7a33ff6bf6efc |
|
BLAKE2b-256 | f80735e08edd77c5a9dd2bfdc5d58fec7eccea3392ce650d90f34230350a496b |