Sling Binary for Windows
Project description
Slings from a data source to a data target.
Installation
pip install sling
Then you should be able to run sling --help
from command line.
Running a Extract-Load Task
CLI
sling run --src-conn MY_PG --src-stream myschema.mytable \
--tgt-conn YOUR_SNOWFLAKE --tgt-object yourschema.yourtable \
--mode full-refresh
Or passing a yaml/json string or file
cat '
source: MY_POSTGRES
target: MY_SNOWFLAKE
# default config options which apply to all streams
defaults:
mode: full-refresh
object: new_schema.{stream_schema}_{stream_table}
streams:
my_schema.*:
' > /path/to/replication.yaml
sling run -r /path/to/replication.yaml
From Lib
Run a replication from file:
import yaml
from sling import Replication
with open('path/to/replication.yaml') as file:
config = yaml.load(file, Loader=yaml.FullLoader)
replication = Replication(**config)
replication.run()
Build a replication dynamically:
from sling import Replication, ReplicationStream
# build sling replication
streams = {}
for (folder, table_name) in list(folders):
streams[folder] = ReplicationStream(mode='full-refresh', object=table_name, primary_key='_hash_id')
replication = Replication(
source='aws_s3',
target='snowflake',
streams=streams,
env=dict(SLING_STREAM_URL_COLUMN='true', SLING_LOADED_AT_COLUMN='true'),
debug=True,
)
replication.run()
Config Schema
--src-conn
/source.conn
and --tgt-conn
/target.conn
can be a name or URL of a folder:
MY_PG
(connection ref in db, profile or env)postgresql://user:password!@host.loc:5432/database
s3://my_bucket/my_folder/file.csv
gs://my_google_bucket/my_folder/file.json
file:///tmp/my_folder/file.csv
(local storage)
--src-stream
/source.stream
can be an object name to stream from:
TABLE1
SCHEMA1.TABLE2
OBJECT_NAME
select * from SCHEMA1.TABLE3
/path/to/file.sql
(if source conn is DB)
--tgt-object
/target.object
can be an object name to write to:
TABLE1
SCHEMA1.TABLE2
Example as JSON
{
"source": {
"conn": "MY_PG_URL",
"stream": "select * from my_table",
"options": {}
},
"target": {
"conn": "s3://my_bucket/my_folder/new_file.csv",
"options": {
"header": false
}
}
}
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
File details
Details for the file sling-windows-amd64-1.2.22.tar.gz
.
File metadata
- Download URL: sling-windows-amd64-1.2.22.tar.gz
- Upload date:
- Size: 53.6 MB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.9.20
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 27ca29f19d9ebd89b4f280875846327a9beedd92b21b8b926f694f5173311b4f |
|
MD5 | 13f251b571526e572ffdb4298d353056 |
|
BLAKE2b-256 | 4f9341e80c1891048fa03c8214b546967438a8107c9954426b13b529603e6131 |