Skip to main content

Slings data from a source to a target

Project description

logo

Slings from a data source to a data target.

Installation

pip install sling

Then you should be able to run sling --help from command line.

Running a Extract-Load Task

CLI

sling run --src-conn MY_PG --src-stream myschema.mytable \
  --tgt-conn YOUR_SNOWFLAKE --tgt-object yourschema.yourtable \
  --mode full-refresh

Or passing a yaml/json string or file

sling run -c '
source:
  conn: MY_PG
  stream: myschema.mytable

target:
  conn: YOUR_SNOWFLAKE
  object: yourschema.yourtable

mode: full-refresh
'
# OR
sling run -c /path/to/config.json

From Lib

from sling import Sling

config = {
  'source': {
    'conn': 'MY_PG',
    'stream': "select * from my_table",
  },
  'target': {
    'conn':  "s3://my_bucket/my_folder/new_file.csv",
  },
}

Sling(**config).run()

Config Schema

--src-conn/source.conn and --tgt-conn/target.conn can be a name or URL of a folder:

  • MY_PG (connection ref in db, profile or env)
  • postgresql://user:password!@host.loc:5432/database
  • s3://my_bucket/my_folder/file.csv
  • gs://my_google_bucket/my_folder/file.json
  • file:///tmp/my_folder/file.csv (local storage)

--src-stream/source.stream can be an object name to stream from:

  • TABLE1
  • SCHEMA1.TABLE2
  • OBJECT_NAME
  • select * from SCHEMA1.TABLE3
  • /path/to/file.sql (if source conn is DB)

--tgt-object/target.object can be an object name to write to:

  • TABLE1
  • SCHEMA1.TABLE2

Example as JSON

{
  "source": {
    "conn": "MY_PG_URL",
    "stream": "select * from my_table",
    "options": {}
  },
  "target": {
    "conn": "s3://my_bucket/my_folder/new_file.csv",
    "options": {
      "header": false
    }
  }
}

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

sling-1.2.4.tar.gz (6.7 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page