`tap-postgres` is a Singer tap for Postgres, built with the Meltano SDK for Singer Targets.
Project description
tap-postgres
Singer tap for Postgres.
Built with the Meltano Singer SDK.
Capabilities
catalog
state
discover
about
stream-maps
schema-flattening
Settings
Setting | Required | Default | Description |
---|---|---|---|
sqlalchemy_url | True | None | Example postgresql://postgres:postgres@localhost:5432/postgres |
ssh_tunnel | False | None | SSH Tunnel Configuration, this is a json object. |
ssh_tunnel.enable | True (if ssh_tunnel set) | False | Enable an ssh tunnel (also known as bastion host), see the other ssh_tunnel.* properties for more details. |
ssh_tunnel.host | True (if ssh_tunnel set) | False | Host of the bastion host, this is the host we'll connect to via ssh |
ssh_tunnel.username | True (if ssh_tunnel set) | False | Username to connect to bastion host |
ssh_tunnel.port | True (if ssh_tunnel set) | 22 | Port to connect to bastion host |
ssh_tunnel.private_key | True (if ssh_tunnel set) | None | Private Key for authentication to the bastion host |
ssh_tunnel.private_key_password | False | None | Private Key Password, leave None if no password is set |
stream_maps | False | None | Config object for stream maps capability. For more information check out Stream Maps. |
stream_map_config | False | None | User-defined config values to be used within map expressions. |
flattening_enabled | False | None | 'True' to enable schema flattening and automatically expand nested properties. |
flattening_max_depth | False | None | The max depth to flatten schemas. |
A full list of supported settings and capabilities is available by running: tap-postgres --about
Configure using environment variables
This Singer tap will automatically import any environment variables within the working directory's
.env
if the --config=ENV
is provided, such that config values will be considered if a matching
environment variable is set either in the terminal context or in the .env
file.
SSH Tunnels (Bastion Hosts)
This tap supports connecting to a Postgres database via an SSH tunnel (also known as a bastion host). This is useful if you need to connect to a database that is not publicly accessible. This is the same as using ssh -L
and ssh -R
, but this is done inside the tap itself.
Installation
-
Developer TODO:
Update the below as needed to correctly describe the install procedure. For instance, if you do not have a PyPi repo, or if you want users to directly install from your git repo, you can modify this step as appropriate.
pipx install tap-postgres
Usage
You can easily run tap-postgres
by itself or in a pipeline using Meltano.
Executing the Tap Directly
tap-postgres --version
tap-postgres --help
tap-postgres --config CONFIG --discover > ./catalog.json
Developer Resources
Initialize your Development Environment
pipx install poetry
poetry install
Create and Run Tests
Create tests within the tap_postgres/tests
subfolder and
then run:
poetry run pytest
You can also test the tap-postgres
CLI interface directly using poetry run
:
poetry run tap-postgres --help
Testing with Meltano
Note: This tap will work in any Singer environment and does not require Meltano. Examples here are for convenience and to streamline end-to-end orchestration scenarios.
Your project comes with a custom meltano.yml
project file already created.
Next, install Meltano (if you haven't already) and any needed plugins:
# Install meltano
pipx install meltano
# Initialize meltano within this directory
cd tap-postgres
meltano install
Now you can test and orchestrate using Meltano:
# Test invocation:
meltano invoke tap-postgres --version
# OR run a test `elt` pipeline:
meltano elt tap-postgres target-jsonl
SDK Dev Guide
See the dev guide for more instructions on how to use the SDK to develop your own taps and targets.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for meltanolabs_tap_postgres-0.0.2.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 758648ec4bfc2deda5711759a7bba72bd5fdf97ab4e80db61ea90a3fd4f1e4ac |
|
MD5 | 9bfac23e1a7ba7037ddf7ef55dbdceae |
|
BLAKE2b-256 | bac2e2a5eec895915386e4efc7527834b88b9ebfedaf435ae1151df914928b69 |
Hashes for meltanolabs_tap_postgres-0.0.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ac58642bf787f609ee0c3da92d11aacc020bd7c11e2248449d3867de9cc0d079 |
|
MD5 | 693ba15a7808ccaa28ec3c0743b23e98 |
|
BLAKE2b-256 | badf9f1cb179763ea78e79bedcd247778ce0e7f9c88cc2b369f278b923d82271 |