instaharvest 2.8.0

Professional Instagram data collection toolkit with automation features

InstaHarvest 🌾

Professional Instagram Data Collection Toolkit - A powerful and efficient library for Instagram automation, data collection, and analytics.

📖 Documentation | 🐛 Report Bug | 💡 Request Feature | 🤝 Contributing | 📋 Changelog

---

✨ Features

• 📊 Profile Statistics - Collect followers, following, posts count
• ✓ Verified Badge Check - Detect if account has verified badge
• 🎭 Profile Category - Extract profile category (Actor, Model, Photographer, etc.)
• 📝 Complete Bio - Extract full bio with links, emails, mentions, and contact info
• 🔗 Post & Reel Links - Intelligent scrolling and link collection (Post vs Reel detection)
• 📥 Media Download - Download Images & Videos (High Quality + yt-dlp support)
• 🎬 Reel Data - Specialized scraping for Reels (views, plays, distinct metrics)
• 🏷️ Tagged Accounts - Extract tags from posts and reels (popup & container support)
• 💬 Comment Scraping - Full comment extraction with likes, replies, author info
• 👥 Followers/Following - Collect lists with real-time output
• 💬 Direct Messaging - Send DMs with smart rate limiting
• 🤝 Follow/Unfollow - Manage following with rate limiting
• ⚡ Parallel Processing - Scrape multiple posts simultaneously
• 🛡️ Graceful Shutdown - Safe interruption (Ctrl+C) with auto-save
• 📑 Excel Export - Real-time data export to Excel
• 🌐 Shared Browser - Single browser for all operations
• 🔍 HTML Detection - Automatic structure change detection
• 📝 Professional Logging - Comprehensive logging system
• 🔔 Notification Reader - Read, filter, and analyze activity feed notifications

---

🚀 Installation

📦 Method 1: Install from PyPI (Recommended) - Click to expand

# Install the package
pip install instaharvest

# Install Playwright browser
playwright install chrome

🔧 Method 2: Install from GitHub (Latest Development Version) - Click to expand

Step 1: Clone the Repository

git clone https://github.com/mpython77/insta-harvester.git
cd insta-harvester

Step 2: Install Dependencies

# Install Python dependencies
pip install -r requirements.txt

# Install Playwright browser
playwright install chrome

Step 3: Install Package in Development Mode (Optional)

# Install as editable package
pip install -e .

OR simply use it without installation:

# ⚡ Run directly from source (Intelligent Session Discovery included)
python examples/save_session.py

---

🔧 Complete Setup Guide

📋 Step-by-Step Setup Instructions - Click to expand

Step 1: Verify Python Installation

# Check Python version (requires 3.8+)
python --version

# Should show: Python 3.8.0 or higher

Step 2: Install InstaHarvest

From GitHub:

git clone https://github.com/mpython77/insta-harvester.git
cd insta-harvester
pip install -r requirements.txt
playwright install chrome

From PyPI:

pip install instaharvest
playwright install chrome

Step 3: Create Instagram Session (REQUIRED!)

Option A: Using Python (Recommended) ⭐

from instaharvest import save_session
save_session()

Option B: Using Example Script

# Navigate to examples directory
cd examples

# Run session setup script
python save_session.py

This will:

1. Open Chrome browser
2. Navigate to Instagram
3. Let you log in manually
4. Save your session to instagram_session.json
5. All future scripts will use this session (no re-login needed!)

Important: Without this session file, the library won't work!

Step 4: Test Your Setup

# First, create your Instagram session (required!)
python examples/save_session.py

# Try the all-in-one interactive demo (recommended for learning)
python examples/all_in_one.py

# Or try production scraping
python examples/main_advanced.py

---

⚠️ IMPORTANT: Always Use ScraperConfig! All examples below use ScraperConfig() for proper timing and reliability. Even when using default settings, explicitly creating config is best practice. This prevents timing issues with popups, buttons, and rate limits. See Configuration Guide for customization options.

🚀 First-Time Setup

Before using any features, create an Instagram session (one-time setup):

from instaharvest import save_session

# Create session - this will open a browser
save_session()

# Follow the prompts:
# 1. Browser will open automatically
# 2. Login to Instagram manually
# 3. Press ENTER in terminal when done
# 4. Session saved to instagram_session.json ✅

That's it! Now you can use all library features. The session will be reused automatically.

---

📖 Quick Start Examples

Example 1: Follow a User - Click to expand

from instaharvest import FollowManager
from instaharvest.config import ScraperConfig

# Create config (customize if needed)
config = ScraperConfig()

# Create manager with config
manager = FollowManager(config=config)

# Load session
session_data = manager.load_session()
manager.setup_browser(session_data)

# Follow someone
result = manager.follow("instagram")
print(result)  # {'success': True, 'status': 'followed', ...}

# Clean up
manager.close()

Example 2: Send Direct Message - Click to expand

from instaharvest import MessageManager
from instaharvest.config import ScraperConfig

# Create config
config = ScraperConfig()
manager = MessageManager(config=config)
session_data = manager.load_session()
manager.setup_browser(session_data)

# Send message
result = manager.send_message("username", "Hello from Python!")
print(result)

manager.close()

Example 3: Collect Followers - Click to expand

from instaharvest import FollowersCollector
from instaharvest.config import ScraperConfig

# Create config
config = ScraperConfig()
collector = FollowersCollector(config=config)
session_data = collector.load_session()
collector.setup_browser(session_data)

# Collect first 100 followers
followers = collector.get_followers("username", limit=100, print_realtime=True)
print(f"Collected {len(followers)} followers")

collector.close()

Example 4: All Operations in One Browser (SharedBrowser) - Click to expand

from instaharvest import SharedBrowser
from instaharvest.config import ScraperConfig

# Create config for better reliability
config = ScraperConfig()

# One browser for everything!
with SharedBrowser(config=config) as browser:
    # Follow users
    browser.follow("user1")
    browser.follow("user2")

    # Send messages
    browser.send_message("user1", "Thanks for the follow!")

    # Collect followers
    followers = browser.get_followers("my_account", limit=50)
    print(f"Followers: {len(followers)}")

Example 5: Scrape Comments from Posts - Click to expand

from instaharvest import CommentScraper
from instaharvest.exporters import export_comments_to_json, export_comments_to_excel
from instaharvest.config import ScraperConfig

config = ScraperConfig()
scraper = CommentScraper(config=config)
session_data = scraper.load_session()
scraper.setup_browser(session_data)

# Scrape comments from a single post
post_url = 'https://www.instagram.com/p/DTLHDJpDAbO/'
comments = scraper.scrape(
    post_url,
    max_comments=100,
    include_replies=True
)

# Access comment data
print(f"Found {len(comments.comments)} top-level comments.")
print("---")

for comment in comments.comments:
    print(f"ID: {comment.id}")
    print(f"User: {comment.author.username}")
    print(f"Text: '{comment.text}'")
    print(f"Time: {comment.timestamp_iso}")
    print(f"Likes: {comment.likes_count}")
    print(f"Reply Count (Extracted): {comment.reply_count}")
    print(f"Nested Replies: {len(comment.replies)}")
    
    if comment.replies:
        for reply in comment.replies:
             print(f"    > Reply ID: {reply.id} | User: {reply.author.username} | Text: '{reply.text}'")
    print("-" * 20)

# Export Options
save_json = True

print("\n--- Exporting Data ---")
if save_json:
    json_filename = f"comments_{comments.post_id}.json"
    if export_comments_to_json(comments, json_filename):
        print(f"[+] Saved JSON to {json_filename}")
    else:
        print(f"[-] Failed to save JSON")


scraper.close()

Example 6: Download Media (Images/Videos) - Click to expand

from instaharvest import SharedBrowser
from instaharvest.config import ScraperConfig

# Create config
config = ScraperConfig()

with SharedBrowser(config=config) as browser:
    # Download from ANY Post or Reel URL
    url = "https://www.instagram.com/reel/C-example..."
    
    # Automatically handles:
    # - Images (High Res)
    # - Videos/Reels (via yt-dlp)
    # - Carousels (multiple files)
    files = browser.download_post(url)
    
    if files:
        print(f"✅ Downloaded {len(files)} files:")
        for f in files:
            print(f"   📂 {f}")

---

📁 Example Scripts

📂 Ready-to-Use Scripts - Click to expand

The examples/ directory contains ready-to-use scripts:

🔑 Session Setup (Required First)

python examples/save_session.py

Creates Instagram session (one-time setup, then reused automatically).

🎮 Interactive Demo

python examples/all_in_one.py

Interactive menu with ALL features:

• Follow/Unfollow users
• Send messages
• Collect followers/following
• Batch operations
• Profile scraping

🚀 Production Scraping

python examples/main_advanced.py

Full automatic profile scraping:

• Collects all post/reel links
• Extracts data with parallel processing
• Exports to Excel + JSON
• Advanced diagnostics & error recovery

🔧 Video & Reel Support (IMPORTANT)

To download or view Videos/Reels correctly, the scraper defaults to using Google Chrome (channel='chrome') instead of the bundled Chromium, as Chromium often lacks necessary video codecs.

Requirements:

• Google Chrome must be installed on your system.
• If you see a "Library Error" regarding Chrome, please install it or switch to channel='chromium' in your config (note: videos might not play/download).

config = ScraperConfig(
    browser_channel='chrome',  # Default: Uses system Chrome for video support
    # browser_channel='chromium' # Use this if you don't need videos
)

⚙️ Configuration Examples

python examples/example_custom_config.py

Shows how to customize configuration (delays, viewport, etc.).

📖 Documentation

📚 Full API Documentation - Click to expand

1. Profile Scraping

from instaharvest import ProfileScraper
from instaharvest.config import ScraperConfig

config = ScraperConfig()
scraper = ProfileScraper(config=config)
session_data = scraper.load_session()
scraper.setup_browser(session_data)

profile = scraper.scrape('username')
print(f"Posts: {profile.posts}")
print(f"Followers: {profile.followers}")
print(f"Following: {profile.following}")
print(f"Verified: {'✓ Yes' if profile.is_verified else '✗ No'}")
print(f"Category: {profile.category or 'Not set'}")
print(f"Bio: {profile.bio or 'No bio'}")
print(f"External Links: {profile.external_links}")
print(f"Threads: {profile.threads_profile}")

scraper.close()

2. Collect Followers/Following

from instaharvest import FollowersCollector
from instaharvest.config import ScraperConfig

# Create config
config = ScraperConfig()
collector = FollowersCollector(config=config)
session_data = collector.load_session()
collector.setup_browser(session_data)

# Collect first 100 followers
followers = collector.get_followers('username', limit=100, print_realtime=True)
print(f"Collected {len(followers)} followers")

# Collect following
following = collector.get_following('username', limit=50)

collector.close()

3. Follow/Unfollow Management

from instaharvest import FollowManager
from instaharvest.config import ScraperConfig

config = ScraperConfig()
manager = FollowManager(config=config)
session_data = manager.load_session()
manager.setup_browser(session_data)

# Follow a user
result = manager.follow('username')
print(result)  # {'status': 'success', 'action': 'followed', ...}

# Unfollow
result = manager.unfollow('username')

# Batch follow
usernames = ['user1', 'user2', 'user3']
results = manager.batch_follow(usernames)

manager.close()

4. Direct Messaging

from instaharvest import MessageManager
from instaharvest.config import ScraperConfig

config = ScraperConfig()
messenger = MessageManager(config=config)
session_data = messenger.load_session()
messenger.setup_browser(session_data)

# Send single message
result = messenger.send_message('username', 'Hello!')

# Batch send
usernames = ['user1', 'user2']
results = messenger.batch_send(usernames, 'Hi there!')

messenger.close()

5. Shared Browser (Recommended!)

Use one browser for all operations - Much faster!

from instaharvest import SharedBrowser
from instaharvest.config import ScraperConfig

# Create config
config = ScraperConfig()

with SharedBrowser(config=config) as browser:
    # All operations use the same browser instance
    browser.follow('user1')
    browser.send_message('user1', 'Hello!')
    followers = browser.get_followers('user2', limit=100)
    profile = browser.scrape_profile('user3')

    # No browser reopening! Fast and efficient!

6. Advanced: Parallel Processing

from instaharvest import InstagramOrchestrator, ScraperConfig

config = ScraperConfig(headless=True)
orchestrator = InstagramOrchestrator(config)

# Scrape with 3 parallel workers + Excel export
results = orchestrator.scrape_complete_profile_advanced(
    'username',
    parallel=3,        # 3 parallel browser tabs
    save_excel=True    # Real-time Excel export
)

print(f"Scraped {len(results['posts_data'])} posts")

7. Post Data Extraction

from instaharvest import PostDataScraper
from instaharvest.config import ScraperConfig

config = ScraperConfig()
scraper = PostDataScraper(config=config)
session_data = scraper.load_session()
scraper.setup_browser(session_data)

# Scrape single post
post = scraper.scrape('https://www.instagram.com/p/POST_ID/')
print(f"Tagged: {post.tagged_accounts}")
print(f"Likes: {post.likes}")
print(f"Date: {post.timestamp}")

scraper.close()

8. Comment Scraping

from instaharvest import CommentScraper
from instaharvest.config import ScraperConfig

# 1. Setup Config & Scraper
config = ScraperConfig(headless=False)  # Set to True for background execution
scraper = CommentScraper(config=config)

# 2. Load Session (Required)
# Ensure you have run 'python examples/save_session.py' first
session_data = scraper.load_session()
scraper.setup_browser(session_data)

# 3. Scrape Comments
result = scraper.scrape(
    'https://www.instagram.com/p/POST_ID/',
    max_comments=100,       # Limit (None = all)
    include_replies=True    # Important: Enable nested reply scraping
)

# 4. Access Data
print(f"Total Comments: {result.total_comments_scraped}")
print(f"Total Replies: {result.total_replies_scraped}")

for comment in result.comments:
    print(f"@{comment.author.username}: {comment.text}")
    print(f"  Likes: {comment.likes_count}")

    # Access Nested Replies
    for reply in comment.replies:
        print(f"    ↳ @{reply.author.username}: {reply.text}")

scraper.close()

Export comments to files:

from instaharvest import export_comments_to_json, export_comments_to_excel

# Export to JSON
export_comments_to_json(comments, 'comments.json')

# Export to Excel
export_comments_to_excel(comments, 'comments.xlsx')

9. Notification Reader

from playwright.sync_api import sync_playwright
from instaharvest import ScraperConfig, StealthManager, NotificationReader
import logging, time

config = ScraperConfig()
logger = logging.getLogger('notif')

with sync_playwright() as pw:
    browser = pw.chromium.launch(headless=False)
    context = browser.new_context(
        storage_state='instagram_session.json',
        viewport={'width': 1280, 'height': 900},
        user_agent=config.user_agent,
    )
    page = context.new_page()

    # Apply stealth
    stealth = StealthManager(config)
    stealth.apply_page_stealth(page)
    page.goto('https://www.instagram.com/', wait_until='domcontentloaded')
    time.sleep(4)

    # Read notifications
    reader = NotificationReader(page, logger, config)
    notifications = reader.read_notifications(max_count=50, scroll=True)

    # Filter by type
    follows = reader.filter_by_type(notifications, 'follow')
    likes = reader.filter_by_type(notifications, 'post_like')

    # Filter by section
    this_week = reader.filter_by_section(notifications, 'This week')

    # Summary statistics
    stats = reader.summary(notifications)
    print(f"Total: {stats['total']}, By type: {stats['by_type']}")

    # Convenience methods
    new_followers = reader.get_new_followers_usernames()

    # Export to JSON-serializable dicts
    data = reader.to_dicts(notifications)

    context.close()
    browser.close()

Supported notification types:

Type	Example
follow	"started following you"
post_like	"liked your post/reel/photo/video"
comment_like	"liked your comment: ..."
comment	"commented: ..."
mention	"mentioned you" / "tagged you"
follow_request	"requested to follow you"
follow_accepted	"accepted your follow request"
thread	"posted a thread you might be interested in"
story	"posted a story"
system	Meta/Instagram system notifications

---

🎯 Complete Workflow Example

🔄 Full Automation Workflow - Click to expand

from instaharvest import SharedBrowser
from instaharvest.config import ScraperConfig

# Create config
config = ScraperConfig()

with SharedBrowser(config=config) as browser:
    # 1. Collect followers from target user
    followers = browser.get_followers('target_user', limit=50)
    print(f"Collected {len(followers)} followers")

    # 2. Follow them
    for follower in followers[:10]:  # Follow first 10
        result = browser.follow(follower)
        if result['success']:  # Check success key
            print(f"✓ Followed {follower}")

    # 3. Send welcome message
    for follower in followers[:5]:
        browser.send_message(follower, "Thanks for following!")

---

📋 Requirements

• Python 3.8+
• Playwright (with Chrome browser)
• pandas
• openpyxl
• beautifulsoup4
• lxml

---

🔧 Session Setup

First-time setup - Save your Instagram session:

Method 1: Using Library Function (Recommended) ⭐

from instaharvest import save_session

# Create session - opens browser for manual login
save_session()

Method 2: Using Example Script

python examples/save_session.py

Both methods will:

1. Open Chrome browser
2. Let you log in to Instagram manually
3. Save session to instagram_session.json
4. All future scripts will use this session (no re-login needed!)

---

📁 Project Structure

🗂️ Package Structure - Click to expand

instaharvest/
├── instaharvest/          # Main package
│   ├── __init__.py        # Package entry point
│   ├── base.py            # Base scraper class
│   ├── config.py          # Configuration
│   ├── profile.py         # Profile scraping
│   ├── followers.py       # Followers collection
│   ├── follow.py          # Follow/unfollow
│   ├── message.py         # Direct messaging
│   ├── post_data.py       # Post data extraction
│   ├── shared_browser.py  # Shared browser manager
│   ├── notifications.py   # Notification reader
│   └── ...                # More modules
├── examples/              # Example scripts
│   ├── example_notifications.py  # Notification reader example
├── README.md              # This file
├── setup.py               # Package setup
└── LICENSE                # MIT License

---

⚙️ Configuration

🛠️ Configuration Options - Click to expand

from instaharvest import ScraperConfig

config = ScraperConfig(
    headless=True,              # Run in headless mode
    viewport_width=1920,
    viewport_height=1080,
    default_timeout=30000,      # 30 seconds
    max_scroll_attempts=50,
    log_level='INFO'
)

---

🛡️ Best Practices

✅ Recommended Practices - Click to expand

1. Use SharedBrowser - Reuses browser instance, much faster
2. Rate Limiting - Built-in delays to avoid Instagram bans
3. Session Management - Auto-refreshes session to prevent expiration
4. Error Handling - Comprehensive exception handling
5. Logging - Professional logging for debugging

---

🔧 Troubleshooting

🔍 Common Issues & Solutions - Click to expand

Installation Issues

Error: "playwright command not found"

# Solution: Install Playwright first
pip install playwright
playwright install chrome

Error: "No module named 'instaharvest'"

# Solution 1: If installed from PyPI
pip install instaharvest

# Solution 2: If using GitHub clone
cd /path/to/insta-harvester
pip install -e .

# Solution 3: Run from project directory
cd /path/to/insta-harvester
python examples/save_session.py  # Works without installation

Error: "Could not find Chrome browser"

# Solution: Install Playwright browsers
playwright install chrome

---

Session Issues

Error: "Session file not found"

# Solution: Create session first (REQUIRED!)
cd examples
python save_session.py

# Then run your script
python all_in_one.py  # or any other script

Data Issues

"Posts: 0" or "Reels: 0" but content exists

If you see 0 posts despite successful scrolling:

1. Ensure you are using the latest version (v2.7.1+)
2. Case Sensitivity: Newest version handles 'Post' vs 'post' automatically.
3. Check save_excel=True output - data might be saved even if console count is confusing.

Error: "Login required" or "Session expired"

# Solution: Re-create session
cd examples
python save_session.py

# Log in again when browser opens

---

Operation Errors

Error: "Could not unfollow @username"

Cause: Unfollow popup appears too slowly for the program

Solution: Increase popup delays in configuration

from instaharvest import FollowManager
from instaharvest.config import ScraperConfig

config = ScraperConfig(
    popup_open_delay=4.0,       # Wait longer for popup
    action_delay_min=3.0,
    action_delay_max=4.5,
)

manager = FollowManager(config=config)

See Configuration Guide for detailed configuration options.

Error: "Could not follow @username"

Solution:

config = ScraperConfig(
    button_click_delay=3.0,
    action_delay_min=2.5,
    action_delay_max=4.0,
)

Error: "Instagram says 'Try again later'"

Cause: Instagram rate limiting - you're doing too much too quickly

Solution: Increase rate limiting delays

config = ScraperConfig(
    follow_delay_min=10.0,      # Wait 10-15 seconds between follows
    follow_delay_max=15.0,
    message_delay_min=15.0,     # Wait 15-20 seconds between messages
    message_delay_max=20.0,
)

---

Slow Internet Issues

Problem: You have slow internet, pages load slowly, getting errors

Solution:

from instaharvest.config import ScraperConfig

config = ScraperConfig(
    page_load_delay=5.0,        # Wait longer for pages
    popup_open_delay=4.0,       # Wait longer for popups
    scroll_delay_min=3.0,       # Slower scrolling
    scroll_delay_max=5.0,
)

# Use with any manager
from instaharvest import FollowManager
manager = FollowManager(config=config)

---

Getting Help

1. Check documentation:

   • README.md - Main guide
   • Configuration Guide - Complete configuration reference
   • Examples Guide - Example scripts guide
   • Changelog - Version history and changes
   • Contributing - How to contribute

2. Common issues:

   • Unfollow errors → Increase popup_open_delay
   • Slow internet → Increase all delays
   • Rate limiting → Increase follow_delay_* and message_delay_*

3. Report bugs:

   • GitHub Issues: https://github.com/mpython77/insta-harvester/issues
   • See CONTRIBUTING.md for bug report guidelines

4. Email support:

   • [e-mail]

---

⚠️ Disclaimer

This tool is for educational purposes only. Make sure to:

• Follow Instagram's Terms of Service
• Respect rate limits
• Don't spam or harass users
• Use responsibly

The authors are not responsible for any misuse of this library.

---

📜 License

MIT License - see LICENSE file for details

---

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

---

📞 Support

• GitHub Issues: Report a bug
• Documentation: Read the docs
• Email: [e-mail]

---

🎉 Acknowledgments

Built with:

• Playwright - Browser automation
• Pandas - Data processing
• BeautifulSoup - HTML parsing

---

Made with ❤️ by Muydinov Doston

Happy Harvesting! 🌾