insta-collect-data: Modular Instagram Data Collector

Project Status: In Development

insta-collect-data is a Playwright-based collector for Instagram hashtags and profiles. It supports photo and carousel posts as well as Reels, extracts post-level engagement metrics, and can compute per-profile engagement rates for influencer research.

The live scraper is functional but depends on Instagram's public markup. Expect occasional breakage when Instagram changes its HTML or tightens its login wall. The HTML comment parser is still experimental.

---

Project Overview

insta-collect-data is a small, modular Python project that automates Instagram in a real browser via Playwright. It is intended for structured data collection for analysis and research: hashtag sampling, creator profiling, engagement comparisons, and small-batch archival work.

---

Current Key Features

• Hashtag Scraping: Collect posts from an explore/tag page.
• Profile Scraping: Collect recent posts for one handle or a list of handles in a single browser session.
• Engagement Metrics: Extract likes, comments, and views (when Instagram exposes them), plus profile followers, following, and total posts.
• Engagement Rate: Compute average engagement per post and ER% for each scraped profile.
• Reels Included: Both /p/ posts and /reel/ links are collected. Older versions filtered videos out; that is no longer the case.
• Rich Post Data: url, username, caption, caption_status, timestamp, is_video, hashtags, mentions, likes, comments, views.
• Session Reuse: Log in once with a visible browser and reuse the saved Playwright profile on later headless runs.
• Login-Wall Handling: Detects Instagram login walls, dismisses cookie/sign-up modals when possible, and gives an actionable error in headless mode.
• Dual Output: Every run writes both JSON and CSV.
• HTML Comment Extraction: Parse comments from a previously saved Instagram HTML page into JSON and XLSX.

---

Installation

pip install insta-collect-data

or install directly from GitHub

pip install git+https://github.com/Alammahadika/insta-collect.git

This distribution installs the insta_collect Python package and both the insta-collect and insta-collect-data CLI commands (same program, two names).

insta-collect-data is the continuation of the earlier insta-collect package (last release 0.1.2) under a new PyPI name.

Requirements: Python 3.10+ and a Chromium browser for Playwright.

Playwright Setup (Required)

Install the browser binaries once after installing the package:

playwright install chromium

First-Time Login (Recommended)

Instagram often blocks logged-out scraping with a login wall. Log in once using a visible browser; the session is stored in the Playwright profile directory (ig_profile by default) and reused afterwards.

insta-collect --user natgeo --limit 6 --headed

Log in manually in the window that opens, then let the run finish. Later runs work headless by default.

Do not commit the ig_profile/ folder — it contains your authenticated session.

---

Usage

The installed CLI is insta-collect (also available as insta-collect-data). It has two modes, and they are mutually exclusive.

Arguments

Argument	Description	Default
--tag	Hashtag to scrape (without #)	—
--user	One handle, or a comma-separated list (natgeo,nasa)	—
--users-file	Text file with one handle per line; # starts a comment	—
--limit	Max posts per target (6–12 is a good range for engagement analysis)	10
--profile	Playwright persistent profile directory (session storage)	ig_profile
--headed	Show the browser window (first-time login / debugging)	headless

Provide either --tag or --user / --users-file.

Hashtag mode

insta-collect --tag donaldtrump --limit 10

Single profile

insta-collect --user natgeo --limit 10

Multiple profiles

insta-collect --user natgeo,nasa --limit 8

Or from a file (handles.txt):

# one handle per line
natgeo
nasa
@somecreator

insta-collect --users-file handles.txt --limit 8

Visible browser

insta-collect --tag donaldtrump --limit 10 --headed

---

Output

Each run writes two files in the current directory.

Mode	Files
Hashtag	result_<tag>.json, result_<tag>.csv
Single profile	result_<handle>.json, result_<handle>.csv
Multiple profiles	result_profiles.json, result_profiles.csv

Post fields (hashtag mode)

[
  {
    "url": "https://www.instagram.com/p/DS3UWMfCRA5/",
    "username": "cdn37",
    "caption": "El presidente de Estados Unidos...",
    "caption_status": "long_text",
    "timestamp": "2025-12-29T22:35:50.000Z",
    "is_video": false,
    "hashtags": ["CDN", "Noticias", "DonaldTrump"],
    "mentions": [],
    "likes": 1200,
    "comments": 34,
    "views": null,
    "source_tag": "donaldtrump"
  }
]

caption_status is one of long_text, short_text, no_text, login_required, or error.

Profile fields (profile mode)

[
  {
    "handle": "natgeo",
    "url": "https://www.instagram.com/natgeo/",
    "followers": 283000000,
    "following": 120,
    "posts_total": 31000,
    "posts_scraped": 8,
    "avg_engagement": 45210.5,
    "engagement_rate_pct": 0.016,
    "posts": [
      {
        "url": "https://www.instagram.com/p/...",
        "username": "natgeo",
        "caption": "...",
        "likes": 40000,
        "comments": 5210,
        "views": null,
        "source_handle": "natgeo"
      }
    ]
  }
]

Engagement rate definition

For each scraped post, engagement is likes + comments, plus views when that post reported a view count. Then:

avg_engagement     = mean(per-post engagement)
engagement_rate_%  = avg_engagement / followers * 100

Instagram displays rounded counts (12K, 1.5M), so parsed metrics are approximations, not exact values.

CSV note

CSV output is flat. In profile mode the nested posts list is written as a single serialized cell per profile. Use the JSON file when you need post-level analysis.

---

How It Works

1. Session resumption: Opens Chromium with the persistent --profile directory, reusing any saved login.
2. Link collection: Navigates to the hashtag or profile page, dismisses the cookie banner, and scrolls until enough /p/ and /reel/ links are found (query strings stripped, duplicates removed).
3. Post extraction: Visits each post, closes the sign-up modal when present, and reads og:description plus page selectors for caption, username, timestamp, and engagement metrics.
4. Filtering: Skips posts that hit a login wall or throw an error. Reels and videos are kept.
5. Aggregation: In profile mode, computes average engagement and ER% from the scraped posts.
6. Export: Writes JSON and CSV.

---

Limitations

• Instagram's login wall is the most common failure. If a headless run reports it, rerun once with --headed and log in.
• Logged-out hashtag grids often expose only Reels; results may be Reel-heavy.
• Metrics are rounded by Instagram (K/M/B) and may be missing entirely.
• Keep --limit modest and expect random delays between requests. This is not a high-volume crawler.
• The persistent profile directory contains credentials; exclude it from version control.

---

Naming Conventions: _ vs - in Python Projects

Context	Use	Example
Package name (PyPI / GitHub)	- (hyphen)	insta-collect-data
Python package directory	_ (underscore)	insta_collect
Python import statement	_ (underscore)	import insta_collect
CLI execution via module	_ (underscore)	python -m insta_collect.cli
Script / file name	flexible	insta-collect.py

---

Collecting Comments from an HTML File

In addition to live scraping, insta-collect-data can parse comments from a saved Instagram HTML page made in a logged-in browser.

This is useful when:

• You already saved the page and want structured comments.
• You want to avoid live requests.
• You need an XLSX deliverable.

Example

cd path/to/insta_collect   # the package directory in a repo checkout
python3 insta-collect.py ~/Downloads/obama.html --preview 20

The script writes instagram_comments.json and instagram_comments.xlsx to the current directory.

Output

[+] Total entries saved: 239
[+] JSON output: instagram_comments.json
[+] XLSX output: instagram_comments.xlsx

Comment fields

• username
• text (comment_text in the README of older releases)
• timestamp when available
• source_file

Notes

• Output filenames are fixed; existing files are overwritten.
• --preview only affects terminal output, not the saved data.
• This feature is experimental and may change.

---

Roadmap

• Live comment scraping.
• XLSX export for live post/profile runs.
• Retry/backoff and proxy support for larger batches.
• Optional pagination beyond the recent grid.

Completed in this release: profile scraping, engagement metrics/ER%, CSV output, Reels support, headless-by-default with persistent login.

---

Ethical Use Notice

This tool is intended strictly for academic research, journalism, and public-interest analysis. Users are responsible for complying with Instagram's Terms of Service and applicable laws.