tap-instagram-user

tap-instagram-user is a Singer tap for the Meta Graph API (Instagram): account-level (user) insights and media (posts) with their per-post insights, built with the Meltano Singer SDK.

About the name: despite the -user suffix (the tap started with the IG User node), it covers multiple Instagram Graph nodes — currently the User node (account insights) and the Media node (posts + per-post insights) — and is structured to add more.

It extracts Instagram insights metrics (views, reach, impressions, ...) day by day, following an ELT strategy: no business transformation is applied by the tap, the raw API response is stored in a raw_data column (JSON), alongside metadata (ig_user_id, metric_name, breakdown_type, metric_date, extraction_date).

Installation

uv sync

Configuration

Main settings

metrics

Each entry defines a metric and generates one stream per metric/breakdown combination (e.g. views + follow_type -> stream ig_user_insights_views_by_follow_type; reach with no breakdown -> ig_user_insights_reach). An entry may override any of the settings above (except access_token/ig_user_id, always global) for itself only:

The stream name also carries metric_type when it is not the default total_value (so a metric can be requested in both types without collision): ig_user_insights_reach (total_value) vs ig_user_insights_reach_time_series. Note that time_series drops breakdowns (an API rule), and to be useful it should be paired with generate_dates_range: inactive (one call returns the whole daily series); with the default total_value + day-by-day you already get a daily series with breakdowns, so time_series is only an opt-in efficiency option for breakdown-less metrics (mainly reach).

{
  "metrics": [
    {
      "metric": "views",
      "breakdowns": ["follow_type,media_product_type", "follow_type", "media_product_type"]
    },
    {
      "metric": "reach",
      "breakdowns": [""],
      "days_to_subtract": 2,
      "start_date": "2026-01-01T00:00:00Z"
    },
    {
      "metric": "impressions",
      "breakdowns": ["media_product_type"],
      "generate_dates_range": "inactive",
      "period": "week",
      "timeframe": "last_30_days"
    }
  ]
}

See config.template.json for a full example.

Media extraction (optional)

In addition to account-level (user) insights, the tap can extract media (posts) and their per-post insights. This is opt-in: media streams are only discovered when media_fields is set, so user-only setups are unaffected.

Two levels, mirroring the Meta API:

• ig_media — the account's media list (GET /{ig_user_id}/media), cursor-paginated. One row per post per day (snapshot), raw post JSON stored in raw_data.
• ig_media_insights_<metric> — per-post insights (GET /{id_post}/insights), one stream per metric/breakdown (child of ig_media). One row per post × metric.

Stream names follow ig_<node>_<edge>[_<metric>[_by_<breakdown>]] (e.g. ig_user_insights_reach, ig_media_insights_views), with the bare node (ig_media) for the object list itself. The explicit edge segment keeps names collision-free as more Meta edges are added (e.g. a future ig_media_comments edge never clashes with the comments insight metric ig_media_insights_comments).

{
  "media_fields": ["id", "timestamp", "media_type", "media_product_type", "like_count", "comments_count"],
  "media_metrics": [
    { "metric": "reach" },
    { "metric": "views" },
    { "metric": "profile_activity", "breakdowns": ["action_type"] }
  ],
  "media_metric_compatibility": {
    "FEED":  ["reach", "views", "likes", "comments", "profile_activity"],
    "REELS": ["reach", "views", "likes", "comments"],
    "STORY": ["reach", "replies", "navigation"]
  }
}

Why media_metric_compatibility is config (not hardcoded): Meta only supports certain metrics per media type, and that vocabulary changes over time. Keeping it in config means a Meta change is fixed by editing config, not by republishing the package. At runtime, a metric is requested for a post only if it is in media_metrics and media_metric_compatibility[<product_type>] — and, when media_metric_compatibility_by_media_type is set, also in media_metric_compatibility_by_media_type[<media_type>] (intersection of the two axes). This second axis captures finer Meta rules the product_type alone can't, e.g. views applies to VIDEO media only, so it is skipped on FEED images. If Meta still rejects a metric both tables claim is valid, the behaviour depends on on_unsupported_metric: fail (default) stops the run loudly (stale-table signal, no silent loss), while skip logs a WARNING and skips just that post/metric (resilient unattended pipelines).

User and media streams are independent — run only user insights by selecting only the ig_user_* streams (media children additionally depend on the ig_media parent at runtime). See select: in meltano.yml.

Configuration via environment variables

Copy .env.example to .env and fill in the real values (never committed). Meltano convention: <PLUGIN_NAME>_<SETTING_NAME> in uppercase, e.g. TAP_INSTAGRAM_USER_ACCESS_TOKEN.

The full list of settings is available via:

tap-instagram-user --about

Usage

Direct CLI (without Meltano)

tap-instagram-user --config config.json --discover > catalog.json
tap-instagram-user --config config.json --catalog catalog.json --state state.json

Via Meltano (recommended)

# Install the Meltano CLI (if not already done)
pipx install meltano

# Install the plugins declared in meltano.yml
meltano install

# Check the config
meltano config tap-instagram-user list
meltano config test tap-instagram-user

# Run the pipeline (extract -> load into Postgres)
meltano run tap-instagram-user target-postgres

Meltano automatically manages state (bookmarks) between runs via its own system database — no need to manually handle a state.json file.

Development

API field/metric reference

docs/meta_api_reference.md lists every field/metric we know per node/edge, validated against the live Meta API (Meta doesn't expose introspection on Instagram nodes, so we test a curated candidate list instead). It's the menu for user_fields/media_fields/metrics, and it surfaces deprecations. For the insights edges it also captures Meta's authoritative live metric list (read straight from the API's error message).

Regenerate it anytime (needs a config.json with access_token/ig_user_id):

python scripts/introspect.py

The git diff of docs/meta_api_reference.{md,yaml} is then the changelog of what Meta added or dropped. Add newly documented items to scripts/api_candidates.yaml when the docs introduce them.

Tests

uv run pytest

Notable implementation details

• Bookmark: based on each day partition's until, capped so it never regresses (notably during the monthly consolidation). A single bookmark per stream (state_partitioning_keys = []), not one per partition.
• Primary key: includes metric_date (the day the data is for) in addition to ig_user_id/metric_name/breakdown_type, to prevent an upsert on the target side from overwriting another day's data.
• Monthly consolidation: on the 1st of the month, the tap automatically re-extracts the entire month before last (Meta insights can still be corrected after publication).
• Media (ig_media / ig_media_insights_<metric>): the per-post insights edge always reports lifetime. The ig_media parent is cursor-paginated and guarded against running twice the same day; the per-metric children are SDK child streams (parent_stream_type) that receive each post's media_product_type and only call metrics declared valid for it.
• Media incremental strategy: by default ig_media is a full daily snapshot. With media_active_window_days set, only the first run backfills (down to media_since); subsequent runs fetch only since = min(last_run, today - window) — a rolling window that refreshes recent posts (whose stats still evolve) and covers any pause, while leaving frozen old posts untouched (so their insights aren't re-fetched). This is the media counterpart of the user-insights monthly consolidation. The driving bookmark is ig_media's extraction_date (presence = backfill done; value = last run).

See the code in tap_instagram_user/streams.py and tap_instagram_user/client.py for details.

SDK Dev Guide

See the Meltano Singer SDK dev guide for more information.