Sling Binary for Mac (AMD64)
Project description
Slings from a data source to a data target.
Installation
pip install sling
Then you should be able to run sling --help
from command line.
Running a Extract-Load Task
CLI
sling run --src-conn MY_PG --src-stream myschema.mytable \
--tgt-conn YOUR_SNOWFLAKE --tgt-object yourschema.yourtable \
--mode full-refresh
Or passing a yaml/json string or file
cat '
source: MY_POSTGRES
target: MY_SNOWFLAKE
# default config options which apply to all streams
defaults:
mode: full-refresh
object: new_schema.{stream_schema}_{stream_table}
streams:
my_schema.*:
' > /path/to/replication.yaml
sling run -r /path/to/replication.yaml
From Lib
Run a replication from file:
import yaml
from sling import Replication
with open('path/to/replication.yaml') as file:
config = yaml.load(file, Loader=yaml.FullLoader)
replication = Replication(**config)
replication.run()
Build a replication dynamically:
from sling import Replication, ReplicationStream
# build sling replication
streams = {}
for (folder, table_name) in list(folders):
streams[folder] = ReplicationStream(mode='full-refresh', object=table_name, primary_key='_hash_id')
replication = Replication(
source='aws_s3',
target='snowflake',
streams=streams,
env=dict(SLING_STREAM_URL_COLUMN='true', SLING_LOADED_AT_COLUMN='true'),
debug=True,
)
replication.run()
Config Schema
--src-conn
/source.conn
and --tgt-conn
/target.conn
can be a name or URL of a folder:
MY_PG
(connection ref in db, profile or env)postgresql://user:password!@host.loc:5432/database
s3://my_bucket/my_folder/file.csv
gs://my_google_bucket/my_folder/file.json
file:///tmp/my_folder/file.csv
(local storage)
--src-stream
/source.stream
can be an object name to stream from:
TABLE1
SCHEMA1.TABLE2
OBJECT_NAME
select * from SCHEMA1.TABLE3
/path/to/file.sql
(if source conn is DB)
--tgt-object
/target.object
can be an object name to write to:
TABLE1
SCHEMA1.TABLE2
Example as JSON
{
"source": {
"conn": "MY_PG_URL",
"stream": "select * from my_table",
"options": {}
},
"target": {
"conn": "s3://my_bucket/my_folder/new_file.csv",
"options": {
"header": false
}
}
}
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
sling-mac-amd64-1.2.20.tar.gz
(59.0 MB
view details)
File details
Details for the file sling-mac-amd64-1.2.20.tar.gz
.
File metadata
- Download URL: sling-mac-amd64-1.2.20.tar.gz
- Upload date:
- Size: 59.0 MB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.9.20
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | ace0164d180f6b4d75cea9e4349dc5f2204d972226b1e69c44e0d8f41547ae99 |
|
MD5 | 37b0704d4a478e079522deec1c3e64aa |
|
BLAKE2b-256 | a6a3bd2b77c06db767bc7755eba681c9d7ae71e1d6174f39e3923c816e74d662 |