Singer.io tap for extracting data from the Twilio API - PipelineWise compatible
Project description
pipelinewise-tap-twilio
Singer tap that extracts data from a Twilio API and produces JSON-formatted data following the Singer spec.
This is a PipelineWise compatible tap connector.
This tap:
-
Extracts the following resources:
- accounts
- addresses
- dependent_phone_numbers
- applications
- available_phone_number_countries
- available_phone_numbers_local
- available_phone_numbers_mobile
- available_phone_numbers_toll_free
- incoming_phone_numbers
- keys
- calls
- conferences
- conference_participants
- outgoing_caller_ids
- recordings
- usage_records
- usage_triggers
- transcriptions
- queues
- message_media
- alerts
-
Extracts TaskRouter resources:
-
Extracts Programmable Chat resources (
members
andchat_messages
areFULL_TABLE
synced, so take care syncing them, they result in a lot of request/data): -
Outputs the schema for each resource
-
Incrementally pulls data based on the input state
Streams
Standard Endpoints:
- Endpoint: https://api.twilio.com/2010-04-01/Accounts.json
- Primary key fields: sid
- Replication strategy: FULL_TABLE
- Transformations: subresources_to_array
- Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Addresses.json
- Parent: account
- Primary key fields: sid
- Replication strategy: INCREMENTAL
- Transformations: subresources_to_array
- Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Addresses/{ParentId}/DependentPhoneNumbers.json
- Parent: addresses
- Primary key fields: sid
- Replication strategy: FULL_TABLE
- Transformations: subresources_to_array
- Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Applications.json
- Parent: accounts
- Primary key fields: sid
- Replication strategy: INCREMENTAL
- Transformations: subresources_to_array
available_phone_number_countries
- Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/AvailablePhoneNumbers.json
- Parent: accounts
- Primary key fields: country_code
- Replication strategy: FULL_TABLE
- Transformations: subresources_to_array
- Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/AvailablePhoneNumbers/{ParentId}/Local.json
- Parent: available_phone_number_countries
- Primary key fields: iso_country, phone_number
- Replication strategy: FULL_TABLE
- Transformations: subresources_to_array
available_phone_numbers_mobile
- Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/AvailablePhoneNumbers/{ParentId}/Mobile.json
- Parent: available_phone_number_countries
- Primary key fields: iso_country, phone_number
- Replication strategy: FULL_TABLE
- Transformations: subresources_to_array
available_phone_numbers_toll_free
- Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/AvailablePhoneNumbers/{ParentId}/TollFree.json
- Parent: available_phone_number_countries
- Primary key fields: iso_country, phone_number
- Replication strategy: FULL_TABLE
- Transformations: none
- Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/IncomingPhoneNumbers.json
- Parent: accounts
- Primary key fields: sid
- Replication strategy: INCREMENTAL
- Transformations: subresources_to_array
- Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Keys.json
- Parent: accounts
- Primary key fields: sid
- Replication strategy: INCREMENTAL
- Transformations: subresources_to_array
- Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Calls.json
- Parent: accounts
- Primary key fields: sid
- Replication strategy: INCREMENTAL
- Transformations: subresources_to_array
- Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Conferences.json
- Parent: accounts
- Primary key fields: sid
- Replication strategy: INCREMENTAL
- Transformations: subresources_to_array
- Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Conferences/{ParentId}/Participants.json
- Parent: conferences
- Primary key fields: uri
- Replication strategy: FULL_TABLE
- Transformations: subresources_to_array
- Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/OutgoingCallerIds.json
- Parent: accounts
- Primary key fields: sid
- Replication strategy: INCREMENTAL
- Transformations: subresources_to_array
- Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Recordings.json
- Parent: accounts
- Primary key fields: sid
- Replication strategy: INCREMENTAL
- Transformations: subresources_to_array
- Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Transcriptions.json
- Parent: accounts
- Primary key fields: sid
- Replication strategy: INCREMENTAL
- Transformations: subresources_to_array
- Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Queues.json
- Parent: accounts
- Primary key fields: sid
- Replication strategy: INCREMENTAL
- Transformations: subresources_to_array
- Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Messages/{ParentId}/Media.json
- Parent: messages
- Primary key fields: sid
- Replication strategy: FULL_TABLE
- Transformations: subresources_to_array
- Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Usage/Records.json
- Parent: accounts
- Primary key fields: account_sid, category, start_date
- Replication strategy: INCREMENTAL
- Transformations: subresources_to_array
- Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Usage/Triggers.json
- Parent: accounts
- Primary key fields: sid
- Replication strategy: INCREMENTAL
- Transformations: subresources_to_array
- Endpoint: https://monitor.twilio.com/v1/Alerts
- Primary key fields: sid
- Replication strategy: INCREMENTAL
- Transformations: none
TaskRouter Endpoints:
- Endpoint: https://taskrouter.twilio.com/v1/Workspaces
- Primary key fields: sid
- Replication strategy: INCREMENTAL
- Transformations: none
- Endpoint: https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/Activities
- Parent: workspaces
- Primary key fields: sid
- Replication strategy: INCREMENTAL
- Transformations: none
- Endpoint: https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/Events
- Parent: workspaces
- Primary key fields: sid
- Replication strategy: INCREMENTAL
- Transformations: none
- Endpoint: https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/Tasks
- Parent: workspaces
- Primary key fields: sid
- Replication strategy: INCREMENTAL
- Transformations: none
- Endpoint: https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/TaskChannels
- Parent: workspaces
- Primary key fields: sid
- Replication strategy: INCREMENTAL
- Transformations: none
- Endpoint: https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/TaskQueues
- Parent: workspaces
- Primary key fields: sid
- Replication strategy: INCREMENTAL
- Transformations: none
- Endpoint: https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/TaskQueues/{ParentId}/CumulativeStatistics
- Parent: task_queues
- Replication strategy: FULL_TABLE
- Transformations: none
- Endpoint: https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/Workers
- Parent: workspaces
- Primary key fields: sid
- Replication strategy: INCREMENTAL
- Transformations: none
- Endpoint: https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/Workers/{ParentId}/Channels
- Parent: workers
- Primary key fields: sid
- Replication strategy: FULL_TABLE
- Transformations: none
- Endpoint: https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/Workflows
- Parent: workspaces
- Primary key fields: sid
- Replication strategy: INCREMENTAL
- Transformations: none
Programmable Chat Endpoints:
- Endpoint: https://chat.twilio.com/v2/Services
- Primary key fields: sid
- Replication strategy: INCREMENTAL
- Transformations: none
- Endpoint: https://chat.twilio.com/v2/Services/{ParentId}/Roles
- Parent: services
- Primary key fields: sid
- Replication strategy: INCREMENTAL
- Transformations: none
- Endpoint: https://chat.twilio.com/v2/Services/{ParentId}/Channels
- Parent: services
- Primary key fields: sid
- Replication strategy: INCREMENTAL
- Transformations: none
- Endpoint: https://chat.twilio.com/v2/Services/{ParentId}/Channels/{ParentId}/Members
- Parent: workspaces
- Primary key fields: sid
- Replication strategy: FULL_TABLE
- Transformations: none
- Endpoint: https://chat.twilio.com/v2/Services/{ParentId}/Channels/{ParentId}/Messages
- Parent: workspaces
- Primary key fields: sid
- Replication strategy: FULL_TABLE
- Transformations: none
- Endpoint: https://chat.twilio.com/v2/Services/{ParentId}/Users
- Parent: services
- Primary key fields: sid
- Replication strategy: INCREMENTAL
- Transformations: none
Authentication
This tap authenticates to the Twilio API using Basic Auth.
To set up authentication simply include your Twilio account_sid
and auth_token
in the tap config.
Quick Start
-
Install
python3 -m venv venv . venv/bin/activate pip install --upgrade pip pip install .
-
Create your tap's
config.json
file. Theapi_key
is available in the twilio Console UI (see Authentication above). Thedate_window_days
is the integer number of days (between the from and to dates) for date-windowing through the date-filtered endpoints (default = 30). Thestart_date
is the absolute beginning date from which incremental loading on the initial load will start.{ "account_sid": "YOUR_ACCOUNT_SID", "auth_token": "YOUR_AUTH_TOKEN", "start_date": "2019-01-01T00:00:00Z", "user_agent": "tap-twilio <api_user_email@your_company.com>", }
Optionally, also create a
state.json
file.currently_syncing
is an optional attribute used for identifying the last object to be synced in case the job is interrupted mid-stream. The next run would begin where the last job left off.{ "currently_syncing": "registers", "bookmarks": { "acounts": "2020-03-23T10:31:14.000000Z", "...": "2020-03-23T00:00:00.000000Z" } }
-
Run the Tap in Discovery Mode This creates a catalog.json for selecting objects/fields to integrate:
tap-twilio --config config.json --discover > catalog.json
See the Singer docs on discovery mode here.
-
Run the Tap in Sync Mode (with catalog)
For Sync mode:
> tap-twilio --config tap_config.json --catalog catalog.json
Messages are written to standard output following the Singer specification. The resultant stream of JSON data can be consumed by a Singer target.
To run tests
- Install python test dependencies in a virtual env and run nose unit and integration tests
python3 -m venv venv
. venv/bin/activate
pip install --upgrade pip
pip install -e .[test]
- To run unit tests:
pytest tests/unit
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for pipelinewise-tap-twilio-1.1.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6b606c00e04346f27e2753d9b2322a4041061e62bc79dc4a570f6e74f2d10737 |
|
MD5 | 7470e7f1654afd62e7a9ffdba1297f3d |
|
BLAKE2b-256 | 602760f78c022a0c24059129d5b318177134dd36157f7c7cc66331bf25f56165 |
Hashes for pipelinewise_tap_twilio-1.1.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 90f6af9d0dd3a50ffe77e580fbad2b0c74085800b0e246350367154e81ad2251 |
|
MD5 | 88341847274e7952c54d75c53f73fda5 |
|
BLAKE2b-256 | e75a37988460929fe78a4afa4557f438436ca6fb02653c3b325f26056edc0ff7 |