Skip to main content tap for extracting CSV files from S3 - PipelineWise compatible

Project description


PyPI version PyPI - Python Version License: MIT

This is a Singer tap that reads data from files located inside a given S3 bucket and produces JSON-formatted data following the Singer spec.

This is a PipelineWise compatible tap connector.

How to use it

The recommended method of running this tap is to use it from PipelineWise. When running it from PipelineWise you don't need to configure this tap with JSON files and most of things are automated. Please check the related documentation at Tap S3 CSV

If you want to run this Singer Tap independently please read further.

Install and Run

First, make sure Python 3 is installed on your system or follow these installation instructions for Mac or Ubuntu.

It's recommended to use a virtualenv:

  python3 -m venv venv
  pip install pipelinewise-tap-s3-csv


  python3 -m venv venv
  . venv/bin/activate
  pip install --upgrade pip
  pip install .


Here is an example of basic config, and a bit of a run down on each of the properties:

    "aws_access_key_id": "ACCESS_KEY",
    "aws_secret_access_key": "SECRET_ACCESS_KEY",
    "start_date": "2000-01-01T00:00:00Z",
    "bucket": "tradesignals-crawler",
    "tables": [{
        "search_prefix": "feeds",
        "search_pattern": ".csv",
        "table_name": "my_table",
        "key_properties": ["id"],
        "delimiter": ","
  • aws_access_key_id: AWS access key ID
  • aws_secret_access_key: AWS secret access key
  • aws_endpoint_url: (Optional): The complete URL to use for the constructed client. Normally, botocore will automatically construct the appropriate URL to use when communicating with a service. You can specify a complete URL (including the "http/https" scheme) to override this behavior. For example
  • start_date: This is the datetime that the tap will use to look for newly updated or created files, based on the modified timestamp of the file.
  • bucket: The name of the bucket to search for files under.
  • tables: JSON object that the tap will use to search for files, and emit records as "tables" from those files.

The table field consists of one or more objects, that describe how to find files and emit records. A more detailed (and unescaped) example below:

        "search_prefix": "exports"
        "search_pattern": "my_table\\/.*\\.csv",
        "table_name": "my_table",
        "key_properties": ["id"],
        "date_overrides": ["created_at"],
        "delimiter": ","
  • search_prefix: This is a prefix to apply after the bucket, but before the file search pattern, to allow you to find files in "directories" below the bucket.
  • search_pattern: This is an escaped regular expression that the tap will use to find files in the bucket + prefix. It's a bit strange, since this is an escaped string inside of an escaped string, any backslashes in the RegEx will need to be double-escaped.
  • table_name: This value is a string of your choosing, and will be used to name the stream that records are emitted under for files matching content.
  • key_properties: These are the "primary keys" of the CSV files, to be used by the target for deduplication and primary key definitions downstream in the destination.
  • date_overrides: Specifies field names in the files that are supposed to be parsed as a datetime. The tap doesn't attempt to automatically determine if a field is a datetime, so this will make it explicit in the discovered schema.
  • delimiter: This allows you to specify a custom delimiter, such as \t or |, if that applies to your files.

A sample configuration is available inside config.sample.json

Based on Stitch documentation

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for pipelinewise-tap-s3-csv, version 1.0.7
Filename, size File type Python version Upload date Hashes
Filename, size pipelinewise_tap_s3_csv-1.0.7-py3-none-any.whl (23.3 kB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size pipelinewise-tap-s3-csv-1.0.7.tar.gz (10.2 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page