tap-dune 0.1.0

Singer tap for extracting data from Dune Analytics API

tap-dune

This is a Singer tap that produces JSON-formatted data following the Singer spec.

This tap:

• Pulls data from the Dune Analytics API
• Extracts data from specified Dune queries
• Produces Singer formatted data following the Singer spec
• Supports incremental replication using query parameters

Installation

pipx install poetry
git clone https://github.com/your-username/tap-dune.git
cd tap-dune
poetry install

Configuration

Accepted Config Options

A full list of supported settings and capabilities is available by running:

poetry run tap-dune --about

Config File Setup

1. Copy the example config file:

   cp config.json.example config.json
   

2. Edit config.json with your settings:

{
    "api_key": "YOUR_DUNE_API_KEY",
    "query_id": "YOUR_QUERY_ID",
    "performance": "medium",
    "query_parameters": [
        {
            "key": "date_from",
            "value": "2025-08-01",
            "replication_key": true
        }
    ],
    "schema": {
        "properties": {
            "day": {"type": "string", "format": "date"},
            "network": {"type": "string"},
            "total_mana": {"type": "number"},
            "total_usd": {"type": "number"}
        }
    }
}

Configuration Fields

Field	Required	Description
api_key	Yes	Your Dune Analytics API key
query_id	Yes	The ID of the Dune query to execute
performance	No	Query execution performance tier: 'medium' (10 credits) or 'large' (20 credits). Defaults to 'medium'
query_parameters	No	Array of parameters to pass to your Dune query
schema	Yes	JSON Schema definition of your query's output fields

Query Parameters

Each query parameter object can have:

• key: Parameter name in your Dune query
• value: Parameter value
• replication_key: Set to true for the parameter that should be used for incremental replication

Schema Configuration

The schema section must define all fields that your Dune query returns. Each field should specify:

• type: The data type ('string', 'number', 'integer', 'boolean', 'object', 'array')
• format (optional): Special format for string fields (e.g., 'date', 'date-time')

Source Authentication and Authorization

1. Visit Dune Analytics
2. Create an account and obtain an API key
3. Add the API key to your config file

Usage

Basic Usage

1. Generate a catalog file:

   poetry run tap-dune --config config.json --discover > catalog.json
   

2. Run the tap:

   poetry run tap-dune --config config.json --catalog catalog.json
   

Incremental Replication

To use incremental replication:

1. Mark one of your query parameters with "replication_key": true
2. Ensure the parameter value is in a format that can be ordered (e.g., dates, timestamps)
3. The tap will track the last value processed and resume from there in subsequent runs

Pipeline Usage

You can easily run tap-dune in a pipeline using Meltano or any other Singer-compatible tool.

Example with target-jsonl:

poetry run tap-dune --config config.json --catalog catalog.json | target-jsonl

Development

Initialize your Development Environment

pipx install poetry
poetry install

Testing

poetry run pytest

SDK Dev Guide

See the dev guide for more instructions on how to use the SDK to develop your own taps and targets.