tap-dune 0.4.0

Singer tap for extracting data from Dune Analytics API

tap-dune

This is a Singer tap that produces JSON-formatted data following the Singer spec.

This tap:

• Pulls data from the Dune Analytics API
• Extracts data from specified Dune queries
• Produces Singer formatted data following the Singer spec
• Supports incremental replication using query parameters
• Automatically infers schema from query results
• Advertises configurable primary keys for correct upsert/dedup behavior in targets

Installation

pip install tap-dune

Configuration

Accepted Config Options

A full list of supported settings and capabilities is available by running:

tap-dune --about

Config File Setup

1. Copy the example config file:

   cp config.json.example config.json
   

2. Edit config.json with your settings:

{
    "api_key": "YOUR_DUNE_API_KEY",
    "query_id": "YOUR_QUERY_ID",
    "performance": "medium",
    "query_parameters": [
        {
            "key": "date_from",
            "value": "2025-08-01",
            "type": "date",
            "replication_key": true,
            "replication_key_field": "day"
        }
    ]
}

Configuration Fields

Field	Required	Description
api_key	Yes	Your Dune Analytics API key
query_id	Yes	The ID of the Dune query to execute
performance	No	Query execution performance tier: 'medium' (10 credits) or 'large' (20 credits). Defaults to 'medium'
query_parameters	No	Array of parameters to pass to your Dune query
schema	No	Optional: JSON Schema definition of your query's output fields. If not provided, schema will be inferred from query results
primary_keys	No	Array of field names that uniquely identify each record. Used by targets for upsert/dedup

Query Parameters

Each query parameter object can have:

• key: Parameter name in your Dune query
• value: Parameter value
• replication_key: Set to true for the parameter that should be used for incremental replication
• replication_key_field: The field in the query results to use for tracking replication state (required if replication_key is true)
• type: The data type of the parameter value. Can be one of:
  • string (default)
  • integer
  • number
  • date
  • date-time

Schema Configuration

The schema can be:

1. Automatically inferred from query results (recommended)
2. Explicitly defined in the config file

When automatically inferring the schema:

• The tap will execute the query once to get sample data
• Data types are detected based on the values in the results
• Special formats like dates and timestamps are automatically recognized
• Null values are handled by looking at other rows to determine the correct type
• If a type cannot be determined, it defaults to string

If you need to explicitly define the schema, each field should specify:

• type: The data type ('string', 'number', 'integer', 'boolean', 'object', 'array')
• format (optional): Special format for string fields (e.g., 'date', 'date-time')

When using incremental replication, the schema configuration is particularly important for the replication key field:

• The field's type in the schema determines how values are compared for incremental replication
• You can specify any type that supports ordering (string, number, integer)
• For date/time fields, you can add the appropriate format ('date' or 'date-time')

Examples of query parameter configurations with different replication key types:

1. Date-based replication (most common):

{
    "api_key": "YOUR_DUNE_API_KEY",
    "query_id": "YOUR_QUERY_ID",
    "primary_keys": ["date", "source"],
    "query_parameters": [
        {
            "key": "start_date",
            "value": "2025-08-01",
            "type": "date",
            "replication_key": true
        }
    ]
}

2. Numeric replication (e.g., for block numbers):

{
    "api_key": "YOUR_DUNE_API_KEY",
    "query_id": "YOUR_QUERY_ID",
    "query_parameters": [
        {
            "key": "min_block",
            "value": "1000000",
            "type": "integer",
            "replication_key": true
        }
    ]
}

3. Timestamp replication:

{
    "api_key": "YOUR_DUNE_API_KEY",
    "query_id": "YOUR_QUERY_ID",
    "query_parameters": [
        {
            "key": "start_time",
            "value": "2025-08-01T00:00:00Z",
            "type": "date-time",
            "replication_key": true
        }
    ]
}

Source Authentication and Authorization

1. Visit Dune Analytics
2. Create an account and obtain an API key
3. Add the API key to your config file

Usage

Basic Usage

1. Generate a catalog file:

   tap-dune --config config.json --discover > catalog.json
   

2. Run the tap:

   tap-dune --config config.json --catalog catalog.json
   

Incremental Replication

To use incremental replication:

1. Mark one of your query parameters with "replication_key": true
2. Ensure the parameter value is in a format that can be ordered (e.g., dates, timestamps, numbers)
3. The tap will track the last value processed and resume from there in subsequent runs

When using incremental replication, you need to configure:

1. The query parameter that will be used for filtering (replication_key: true)
2. The field in the query results that will be used for state tracking (replication_key_field)
3. The data type of the parameter (type)

For example, if your query:

• Takes a date_from parameter for filtering
• Returns records with a day field containing dates
• You want to use that day field for tracking progress

Your configuration would look like:

{
    "query_parameters": [
        {
            "key": "date_from",
            "value": "2025-08-01",
            "type": "date",
            "replication_key": true,
            "replication_key_field": "day"
        }
    ]
}

The tap will:

1. Use date_from to filter the query results
2. Track the day field values from the results
3. Use those values to set date_from in subsequent runs

The parameter type can be:

• date or date-time for date-based parameters
• integer or number for numeric parameters
• string (default) for text parameters

Pipeline Usage

You can easily run tap-dune in a pipeline using Meltano or any other Singer-compatible tool.

Example with target-jsonl:

tap-dune --config config.json --catalog catalog.json | target-jsonl

When loading to a database target that performs upserts (e.g., Snowflake):

• Set primary_keys in the tap config to the fields that uniquely identify a row in your query output (e.g., ["date", "source"]).
• Ensure your loader configuration (e.g., PipelineWise or Meltano target) uses the same primary keys for merge/upsert.
• For append-only behavior, leave primary_keys empty and configure your loader for pure inserts.

Development

Initialize your Development Environment

# Clone the repository
git clone https://github.com/blueprint-data/tap-dune.git
cd tap-dune

# Install Poetry
pipx install poetry

# Install dependencies
poetry install

Development Workflow

This project follows Semantic Versioning and uses Conventional Commits for automatic versioning.

1. Create a feature branch:

   git checkout -b feat/your-feature
   # or
   git checkout -b fix/your-bugfix
   

2. Make your changes and commit using conventional commits:

   # For new features
   git commit -m "feat: add new feature X"
   
   # For bug fixes
   git commit -m "fix: resolve issue with Y"
   
   # For breaking changes
   git commit -m "feat: redesign API
   
   BREAKING CHANGE: This changes the API interface"
   

   Commit types:

   • feat: A new feature (minor version bump)
   • fix: A bug fix (patch version bump)
   • docs: Documentation only changes
   • style: Changes that don't affect the code's meaning
   • refactor: Code change that neither fixes a bug nor adds a feature
   • perf: Code change that improves performance
   • test: Adding missing tests
   • chore: Changes to the build process or auxiliary tools
   • BREAKING CHANGE: Any change that breaks backward compatibility (major version bump)

3. Run tests:

   poetry run pytest
   

4. Create a pull request to main

Release Process

1. Create a release branch from main:

   git checkout main
   git pull
   git checkout -b release
   

2. Push the branch:

   git push -u origin release
   

3. The release workflow will automatically:

   • Analyze commits since last release
   • Determine the next version number based on commit types:
     • fix: → patch version (1.0.0 → 1.0.1)
     • feat: → minor version (1.0.0 → 1.1.0)
     • BREAKING CHANGE: → major version (1.0.0 → 2.0.0)
   • Update CHANGELOG.md
   • Create a git tag with the new version
   • Create a GitHub release
   • Build and publish to PyPI

   Note: Only commits following the Conventional Commits format will trigger version updates.

4. After successful release:

   • Create a PR from the release branch to main
   • This PR will contain all the version updates (CHANGELOG.md, version number)
   • Merge to keep main up-to-date with the latest release
   • Note: Only blueprint-data team members can merge to main

5. Clean up:

   git checkout main
   git pull
   git branch -d release
   

Repository Permissions

This repository follows these security practices:

• Only blueprint-data team members can merge to main
• All PRs require at least one review
• All tests must pass before merging
• Branch protection rules prevent bypassing these requirements

Testing

poetry run pytest

SDK Dev Guide

See the dev guide for more instructions on how to use the SDK to develop your own taps and targets.